Use openscad_offliner for offline documentation

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Use openscad_offliner for offline documentation

runsun
This post was updated on .
Just finished a script that allows users to download the entire documentation from the wiki for offline reading.

All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are stripped off.

Here are 2 screenshots:


It has only one file. You can find it in the repo: https://github.com/runsun/openscad_offliner

Require: python 2.7, BeautifulSoup 4.0 above

Usage:
1) Save this file in folder x.
2) In folder x, type:

python openscad_offliner.py

or to save log to a file:

python openscad_offliner.py > openscad_offliner.log

All web pages will be saved in x/openscad_docs,
and all images in x/openscad_docs/imgs

It will generate 94 pages (1.3 MB) + 102 images (1.4 MB).

$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

doug.moen
Looks pretty good.

On 7 July 2015 at 21:10, runsun <[hidden email]> wrote:
Just finished a script that allows users to download the entire documentation from the wiki for offline reading.

All the non-OpenSCAD-related stuff (like wiki menu, wiki warning, etc) are stripped off.

Here are 2 screenshots:


It has only one file. You can find it in the repo: https://github.com/runsun/openscad_offliner

Require: python 2.7, BeautifulSoup 4.0 above

Usage:
1) Save this file in folder x.
2) In folder x, type:

python openscad_offliner.py

or to save log to a file:

python openscad_offliner.py > openscad_offliner.log

All web pages will be saved in x/openscad_docs,
and all images in x/openscad_docs/imgs

It will generate 94 pages (1.3 MB) + 102 images (1.4 MB).

$ Runsun Pan, PhD
$ -- OpenScad_DocTest: doc and unit test ( Github, Thingiverse )
$ -- hash parameter model: here, here
$ -- Linux Mint 17.1 Rebecca x64 + OpenSCAD 2015.03.15/2015.04.01.nightly


View this message in context: Use openscad_offliner to download online doc
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
It is possible to :

1) Write a shell script to zip the folder
2) Use a cron job to download and zip it nightly
3) Then put it in the main repo.

This way, the wiki page becomes the only source of doc we need to maintain.


$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

jdawgaz
If this works, and I will check if it does a little later.
If it does, and includes the cheat sheet, then maybe a future version of openscad could have an optional Environment Variable to point to the docroot (like file:///blahblah/), or if no env variable found, what it points to now.




Jerry

--
Extra Ham Operator: K7AZJ
Registered Linux User: 275424
Raspberry Pi and Arduino developer

The most exciting phrase to hear in science - the one that heralds new discoveries - is not "Eureka!" but "That's funny...".
- Isaac. Asimov

If you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime. 
- Anonymous

If writing good code requires very little comments, then writing really excellent code requires no comments at all!
- Ken Thompson


On Tue, Jul 7, 2015 at 11:46 PM, runsun <[hidden email]> wrote:
It is possible to :

1) Write a shell script to zip the folder
2) Use a cron job to download and zip it nightly
3) Then put it in the main repo.

This way, the wiki page becomes the only source of doc we need to maintain.






-----

$  Runsun Pan, PhD

$ -- OpenScad_DocTest: doc and unit test ( Github , Thingiverse  )

$ -- hash parameter model: here , here

$ -- Linux Mint 17.1 Rebecca x64  + OpenSCAD 2015.03.15/2015.04.01.nightly




--
View this message in context: http://forum.openscad.org/Use-openscad-offliner-to-download-online-doc-tp13096p13098.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

tp3
The scraped HTML would need quite some additional clean-up

* it fires of a number of JavaScripts that access the Wikibooks site
* all the CSS files are loaded online
* most of the images don't work (at least in Firefox) as they have a
   srcset attribute (which I did not even know until just now)
* the index looks strange due to missing SVG files
* the chapter links into other pages don't work
* the edit links are displayed as []

I guess all that is fixable, but I do wonder how much of a moving
target this is chasing as it's trying to handle the output of
another generator which could change with every Wikimedia release.
It might still be a good idea to at least have a look at the
export option and render the original data.

But then if the script is maintained and updated to follow the
changes, that might not be a huge issue.

ciao,
   Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
@Torsten,

Thx for pointing out those issues. I'll get them fixed.

tp3 wrote
* the index looks strange due to missing SVG files
* the chapter links into other pages don't work
Where are index and chapter links ? Maybe because thatI have
css files linked to the web(which should be fixed) so that I don't
see any problem ?

tp3 wrote
I guess all that is fixable, but I do wonder how much of a moving
target this is chasing as it's trying to handle the output of
another generator which could change with every Wikimedia release.
It might still be a good idea to at least have a look at the
export option and render the original data.

But then if the script is maintained and updated to follow the
changes, that might not be a huge issue.

ciao,
   Torsten.
I think before we figure out a better way to solve the burden of
having to maintain several sources of doc, this might serve as
a temporary solution.
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
In reply to this post by tp3
Btw, the [edit] => [] is intentional. I figure that there's no point of editing an offline wiki doc ('cos it needs wiki's editing feature) so I remove the word "edit".

tp3 wrote
* the edit links are displayed as []
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

tp3
In reply to this post by runsun
On 07/12/2015 02:21 AM, runsun wrote:
> Where are index and chapter links ? Maybe because thatI have
> css files linked to the web(which should be fixed) so that I don't
> see any problem ?
>
I mean the file://document#chapter links. The #chapter part seems
to be missing everywhere (maybe due to the link rewriting?)

Example:

file://.../openscad_docs/OpenSCAD_User_Manual

3.2 Primitive Solids - cube, sphere, cylinder & polyhedron

The "cylinder" link should go to Primitive_Solids#cylinder to
jump directly into that chapter.

> Btw, the [edit] => [] is intentional. I figure that there's no point of
> editing an offline wiki doc ('cos it needs wiki's editing feature) so I
> remove the word "edit".
>
Ah, all the remaining [] looked a bit strange. If it's easy, it
would look better if those are removed completely. But that's
just minor cosmetics.

Talking cosmetics... :-) is it possible to add .html to the
files? It seems to work fine without, though.

> I think before we figure out a better way to solve the burden of
> having to maintain several sources of doc, this might serve as
> a temporary solution.
>
I agree, it's much better than having different sources and it's
the only working solution to get a usable off-line documentation
right now.

Reading https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content
it looks like the footer should also be extended a bit to clarify
the license. I currently think it's best to use the CC-BY-SA, but
I'll have a look at the GNU-FDL too.
Using CC-BY-SA would mean to also add a license link and a short
info that the content was modified (maybe something like "page
modified for off-line reading" is sufficient?).

ciao,
   Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
All issues are fixed (not update in repo yet), except the license part. Please let me know what's the best description.

Will do more offline tests to make sure before I push it to the repo.

Also, if this comes to being usable, maybe consider if there's a better way of using it. No matter how I'll write down clearly development instruction in case someone wants to modify it.
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
This post was updated on .
Just pushed it to the openscad_offliner repo. It seems working fine with offline test on Chrome and Firefox.  

Here are the fixes to tp3's findings :
* it fires of a number of JavaScripts that access the Wikibooks site ==> ALL javascripts are shutdown

* all the CSS files are loaded online ==> All needed CSS files are downloaded. There is still at least one page (Positioning_an_object) trying to link to remote css, but it's not harmful ('cos all required css are already downloaded) and hard to clean out (they seem to be retrieved by javascript), so I leave them there.

* most of the images don't work (at least in Firefox) as they have a
   srcset attribute (which I did not even know until just now) ==> all srcset attributes removed

* the index looks strange due to missing SVG files  ==> fixed

* the chapter links into other pages don't work ==> fixed

* the edit links are displayed as [] ==> all [edit] removed

* is it possible to add .html to the files? It seems to work fine without, though. ==> added
To-do:
* licensing statement
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
Just updated openscad_offliner:

1) All styles are taken care of. So there is no more attempt of internet connection;

2) Code has gone thru major refacturing into smaller pieces for easier maintenance .

3) license statement now looks like this (do let me know if this is good 'cos I simply copy / modify a statement from the wiki licensing page without giving much thought):

Content of this page is extracted on 2015/07/14 11:35 from the online OpenSCAD Wikipedia article OpenSCAD_User_Manual (released under the Creative Commons Attribution-Share-Alike License 3.0) using openscad_offliner
I'd say the first stage of an offliner is completed. I'll leave it for now (except bug fixes) until further attention comes up.
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Use openscad_offliner to download online doc

runsun
Updated on 2016, March 20th:

* Major fix for bug believed to be a cross-platform issue. The original version, which was developed in Linux, is saved as openscad_offliner_2015.py. It is made to work in Windows in the current version.
* Include openscad_docs.zip. Users don't need to run the python script, unless a most recent copy is needed.
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets