Have been working on the compute cluster build for Astronomy projects.
Have tried CloudStack, and the networking did me in. Long story short, that is a laser gun for the fly I need to conquer.
Then I managed to get Ambari up and running. Again, it is a little too heavy for what I want to start with. It was a great experience to see how it all comes together under one pane of glass. However, I think I have a different idea that will not require so much overhead. It is a slick way to deploy a stack though. Along the way, I discovered vagrant, and that alone was worth the trouble to get working, Makes spinning up VM easy peasy. I will probably make it a little more customized with each iteration I work on. And, I can keep the old setups around for re-use if I need them.
I managed to get Jupyter installed without too much trouble. I had installed a bunch of R and Python libraries on the Zeppelin VM, so I am hoping that I can get the workstation configured the same, and then use the cluster simply for compute. I really want to minimize the admin tasks going on in the VMs since I am so short of memory. I can probably find a refurb server with everything I need before I can afford the DDR3 DIMMs to beef it up. Womp womp...
Today, I want to play with the Vagrant bridge config to get the VMs on the main network, which should allow me to get Hadoop and Spark spanning all the machines, and leave the VMs with just Hadoop clients and Spark agents running. That will take a little sorting with Hadoop first.
I couldn't sleep last night, so I started the install of some software to analyze Chandra X-Ray data. That looks to be pretty interesting. The last file was about 1G, and it was timing out, so I stopped at that point and went to bed.
This morning when I got up, I finished the Chandra Ciao installation. It passed the automated tests, but is not happy with my interactive shell. I just need to adjust some environment variables, and it should be good to go.
I did skip the programming lesson from Coursera on the Data Driven Astronomy weekly syllabus. It was getting too heavy on the programming aspects, so I am curious to see how it wraps up this week and next. I am still going to do the videos and work through the rest though. I also waited too late in the week to start. I knew it was going to be involved, and I let it slip, so my bad.