WGET: download an HTTP or FTP directory recursively

Many science data sites are an Apache server with a HTTP index.html file listing, perhaps recursively, a lot of files and directories. You can download them recursively to your PC as follows (use this responsibly, you can consume gigabytes of data at cost to science host or yourself!)


wget -r -np -nc -nH --cut-dirs=4 --random-wait --wait 1 -e robots=off http://mysite.com/aaa/bbb/ccc/ddd/

This plops the files to whatever directory you ran the command in.

To use this on FTP site, just change the http:// to ftp:// and type the proper ftp site address.

Option explanation:
-r: download recursively (and place in recursive folders on your PC)
-np: Never get parent directories (sometimes a site will link back up and you don’t want that)
-nc: no clobber — don’t re-download files you already have
-nH: don’t put obnoxious site name directories on your PC
–cut-dirs=4: don’t put an obnoxious hierarchy of directories above the desired directory on your PC. Note you must set the number equal to the number of directories on server (here aaa/bbb/ccc/ddd is four)
-e robots=off: Many sites will block robots from mindlessly consuming huge amounts of data. Here we override this setting telling Apache that we’re (somewhat) human.
–random-wait: To avoid excessive download requests (that can get you auto-banned from downloading) we politely wait in-between files–better than trying to get yourself un-banned!
–wait 1: making the random wait time average to about 1 second before starting to download the next file.

Setting up GCX (Linux astrometry program)

The first step is sudo apt-get install gcx

you will need the GSC ACT catalog, downloaded using wget. There are a lot of directories with lots of ~50KB files, so it will take a while even with a very fast internet connection. Maybe there is a better way to use wget to download a huge amount of small files

mkdir ~/gsc-act && cd ~/gsc-act
wget -r -nH --cut-dirs=5 ftp://cdsarc.u-strasbg.fr/pub/cats/I/255/GSC_ACT/

set File>File and Device Options/GSC to /home/username/gsc-act
where username is your linux login
—————————-
You will need the Tycho2 star catalog, downloaded using wget.


mkdir ~/tycho2 && cd ~/tycho2
wget -r -nH --cut-dirs=5 ftp://cdsarc.u-strasbg.fr/pub/cats/I/259/

Once you see that they’ve downloaded correctly, do:

zcat tyc2.dat.??.gz >tycho2.dat

set File>File and Device Options/Tycho2 to /home/username/tycho2
where username is your linux login

Setting up Astrometry.net program

Note: if you have Ubuntu 14.04, you can simply type: sudo apt-get install astrometry.net
Otherwise, or if you want the latest version, follow the steps below.

If you want a nice wrapper program I wrote that plots right ascension / declination and azimuth /elevation for your image, check out:
https://github.com/scienceopen/astrometry the fits2azel.py is what you want.

setup the prereqs:
sudo apt-get install libcairo2-dev libnetpbm10-dev netpbm \
libpng12-dev libjpeg-dev python-numpy \
python-pyfits python-dev zlib1g-dev \
swig cfitsio-dev

then download newest .bz2 file from: http://astrometry.net/downloads/
extract it to your ~ directory, then go in and type in Terminal
make
make py
make extra
sudo make install

edit your .bashrc file, adding the line
export PATH="$PATH:/usr/local/astrometry/bin"

to increase performance, edit
sudo nano /usr/local/astrometry/etc/astrometry.cfg
(sometimes it’s in /etc/astrometry.cfg)
and uncomment the
inparallel
and modify
add_path /opt/astrometry/data
note: TILDE won’t work! actually use /home/username
where username is your Linux login.

now,
sudo mkdir -p /opt/astrometry/data
for i in {08..19}; do
wget -nc -nd -P /opt/astrometry/data http://broiler.astrometry.net/~dstn/4200/index-42"$i".fits
done

Ubuntu: Fixing Desktop icons and Menu items for MATLAB

MATLAB doesn’t automatically setup itself in the menus of Ubuntu, but you can easily fix this with these two steps:

type in Terminal:
sudo wget http://upload.wikimedia.org/wikipedia/commons/2/21/Matlab_Logo.png -O /usr/share/icons/matlab.png

sudo nano /usr/share/applications/matlab.desktop

cut and paste these lines:
#!/usr/bin/env xdg-open
[Desktop Entry]
Type=Application
Icon=/usr/share/icons/matlab.png
Name=MATLAB R2014a
Comment=Start MATLAB - The Language of Technical Computing
Exec=matlab -desktop
Categories=Development;

Of course, changing MATLAB R2014a to the version you have (not critical)

(based off of: https://help.ubuntu.com/community/MATLAB/)

Installing GNU Radio in Ubuntu 12.04

This blog page is now obsolete, you should use the original reference:
http://gnuradio.org/redmine/projects/gnuradio/wiki/InstallingGR#Using-the-build-gnuradio-script

This procedure shows how to install GNU Radio on Ubuntu 12.04. this process may take 1-2 hours if you have a slower system and/or internet connection. Part of this process will be downloading large packages that you may not already have installed.
1) mkdir ~/gnuradio
1.1) cd ~/gnuradio
2) wget http://www.sbrac.org/files/build-gnuradio && chmod a+x ./build-gnuradio && ./build-gnuradio –verbose
3) Proceed? Y
4) Sudo privilidges? Y
Now you will see a lot of text whizzing by as the build-gnuradio script checks for prerequisite packages and downloads missing packages. Since you used the –verbose option, you’ll see a percentage complete indicator as the script works to download packages.
Once installation is complete, type in Terminal:
gnuradio-companion
to run the graphical programming interface of GNU Radio Companion. You can press F6 to run the code.