Over the past weeks, I have been working with the Open Mainframe Project (OMP) as a part of their summer internship program. It was a wonderful opportunity and I got to work on “dope” tech like “Docker” & “Kubernetes“. The experience was amazing and I learned a lot throughout the program. In this post, I have decided to talk about the organization, the internship program, and what I have done in the past 10 weeks. I hope this blog post will be helpful to those interested in the OMP or my work or those interested to apply for this internship program in the future.
It all started with an interest in ‘Linux’, ‘Infra’ & ‘Open Source’. My last summer internship with Pharo Consortium was my first ever internship and moreover, it was remote, Open Source & associated with Google Summer of Code. While I thought there is no program similar to this, I found the Open Mainframe Project Internship Program. I couldn’t resist apply for it, besides I really wanted to contribute to an Open Source project this summer.
Initially, when I was going through the list of projects, most of the projects were legacy projects. Although I am fairly familiar with the tech in the legacy projects, I wasn’t sure if I should apply for those projects. I did some amount of research to understand the projects on the forums, Github repositories, etc. In fact, I tried emailing some of the past interns. And before I understood the legacy projects completely, new projects were added to the list. Guess what? I found a project which I am some what familiar with and involving the tech I love. The project is titled “Document the deployment of Kubernetes cluster on IBM LinuxONE”. I was lucky to be familiar with ‘Kubernetes’. I have worked on it as a part of a course. This was a big deal breaker for me. It helped me write a good proposal & get through the phone interview (I will talk more about the selection procedure in the following sections). To be frank, it took me about 6 months to get a hang of what Kubernetes is and how it works. And what I learned then was just an overview. Through this program, I learned a bit more. And frankly, there is still a lot more to learn 🙂 .
Open Mainframe Project (OMP)
Talking about the organization, remember the old mainframe systems? Did you know they have evolved and are being used by enterprises currently?
Yes, you heard it right. If you don’t believe me, go take a look at their website. The modern Mainframe has a lot of benefits over their alternatives. Some of them are
- Security – They offer 100% encryption to protect your data from internal and external threats by employing a dedicated cryptographic coprocessor, the CP Assist for Cryptographic Function (CPACF).
- Availability – They deliver record uptime and 99.999% availability.
- Performance – With the industry’s fastest commercially available processor at 5Ghz, they can host a 17TB MongoDB database on a single system with 2.4x better throughput.
- Scalability – They are designed to support thousands of virtual machines besides the ability to serve vast amounts of data, and provides real-time insight and scalable sub-second transactional capability. They can support up to 2M Docker containers.
So where does the Open Mainframe Project come into the picture?
Open Mainframe Project is an Open Source project under the Linux Foundation, aimed at eliminating barriers to Open Source adoption on the Mainframe. The project is intended to serve as the focal point for deployment and use of Linux and Open Source in a mainframe computing environment. In a way, the project also demonstrates the value of mainframe on technical and business levels. In simple terms, the project develops tools & resources to support Linux & various Open Source technologies on the mainframe, besides creating a community of Mainframers and expanding the reach to academic institutions to assist in educating mainframe engineers of tomorrow.
In short, Open Mainframe Project = Mainframe ♥️ Open Source Software
Summer Internship Program
So, the OMP summer internship program is fairly new. It was started in 2016 and the usual intake is around 8-10 students. The organization usually puts out a list of project ideas which can be chosen by the students along with the contact details of the mentor(s). There’s also an option for proposing a new idea which should involve development activity for Linux & Open Source on Mainframe. If the idea gets accepted, the organization finds a suitable mentor for the project.
The only prerequisite for the program is that you need to be a student & the procedure for applying is not so complex. There are just two steps involved in the application.
- Writing a proposal
- 1-1 phone interview
The first step is to write a proposal, which should include details of the project, project timeline, development methodology, previous open source/ industry experience, links to open source contributions (if any), one or two references. All the things required in the proposal can be found on the internship page. The proposal need not be very long but should be precise and simple. From my personal experience, I have a few tips for writing a good proposal
- Divide the proposal into sections based on what all should be included.
- Use a table for the timeline. Timeline can be based on days or weeks. Prefer to plan according to each week.
- It is nice to include buffer periods & review periods in the timeline.
- Use images, or something like Coggle to explain the project.
- Get a review from the mentor before submitting it. By mentor, I mean the project mentor. It is not always that mentors will be willing to review, as they would like to fairly judge all the proposals. So, ask if would be willing to review before sending it.
- If you know someone who has worked on tech involved in the project or who have previously participated in these open source programs, try to get a review from them too. There’s always a chance for improvement.
- Use Google docs or similar collaborating platforms, to write the proposal. It makes it easy to get it reviewed, or for anyone to give suggestions or comments.
- For references, give the details of one or two people who would be willing to refer you, in the proposal.
- List all the open source contributions you have done in the past. Contributions are not limited to code contributions. They can any thing such as improving documentation, participating in discussions, creating issues, etc.
After submitting the proposal, there will be an initial filtering based on the proposals. The mentor(s) of the respective projects will schedule a call and will interview the student applicants. This helps in getting a good understanding of the applicants. In my case, it was more of a discussion the project rather than an interview with questions. It was a good experience for me, my mentor and I had a WhatsApp call where we discussed about the internals of the project, which documentation tool to use, where to make changes in my proposal, where to start, what all should be achieved by the end of the project, etc. After the phone interview, in a few weeks, the selected students will be informed and the same will be announced publicly through a blog post.
If you are a student and if you love Linux & Open Source, I would highly recommend this program. Whatever may be the end result, I would recommend applying for this program. You learn a lot in the process of writing the proposal. Knowledge never goes to waste. I would say applying for the program is one type of experience and the actual program is another type of experience.
I guess I didn’t talk about the stipend. Yes, this is a paid program and every intern will be paid decently. Besides the stipend, every intern will get an opportunity to present the work done in an industry conference. The travel and accommodation costs for the conference will be covered by OMP.
Coming to the work I did, it involved a lot of stuff ranging from setting up the Linux environments, installing stuff (this was a lot), connecting multiple systems, testing the setup, to going down a rabbit hole at times, documenting things, making videos on the same, and a lot of troubleshooting. By ‘Troubleshooting’ here, I mean debugging systems to understand why setup of a component is failing or why two components are not communicating, understanding logs and so on. Of all the things I learned, I think ‘troubleshooting’ is something that was not easy and also fun to do. I did so much of troubleshooting that, I kind of remember some of the bugs & the solutions 😅. I can sum up my work as ‘setting up stuff involving lots of bugs & errors’.
Jokes apart, my work was to setup Kubernetes clusters on IBM tested and supported Linux distributions such as RHEL, SLES & Canonical. I had to set up all these on LinuxOne machines, which are a part of the mainframe series created by IBM that runs Linux. More specifically, I used the LinuxOne Community Cloud to set up the cluster. By the end of the program, I have managed to deploy Kubernetes clusters on SLES (SLES 12SP3) & Canonical (Ubuntu 18.04). While deployment on RHEL was done by co-intern Asish Varanasi.
Initially, I started off with setting up KVM’s on a single SLES SP3 instance with the help of my mentor. I installed Ubuntu 18.04 on the KVM instances and started to deploy the cluster. As a base reference, I followed ‘Kubernetes the hard way‘ by Kelsey Hightower & the ‘Kubernetes cluster step by step series‘ by Igor Cicimov. Although I found the KVM’s to be slow I went ahead with the cluster setup. It was taking a very long time for ‘kubectl’ commands to return the output. I tried analyzing the problem with the help of my mentor and ended up with a gist. My mentor had a talk with the LinuxOne Community Admin and increased our instance’s RAM because of this issue. But, it was still slow. Soon, we realized that the LinuxOne Community Cloud itself is running on a hypervisor and moreover, we have set up a KVM hypervisor on top of it. So, the nested virtualization was the problem here. In fact, my co-intern who was working on RHEL was not able to setup docker because of the virtualization.
In the next few days, my mentor asked me to change the setup. He asked me to use each LinuxOne Community Cloud instance as one node. And this worked. When compared to the previous setup, it was incredibly fast. With this setup, I was able to complete the work smoothly and also at a faster pace. Soon, I finished off the documentation and jumped into testing the setup. I was creating multiple docs as there were multiple ways to setup things. And I didn’t want to delete some of them, therefore I pushed all of them to the same Github Repo. As I was using the asciidoc tool, I was able to generate the docs into various output formats. So, I tried hosting the docs, using Github Pages by generating the docs as html. Moreover, Github automatically renders asciidoc files when we try to view them. This made everything easier. Maybe this is an example of ‘how Open Source makes things easy’ 😛 .
In my final set up, I was able to deploy a 3 node cluster, with 1 node as Master & 2 nodes as Workers. I used Docker as the container run-time, Etcd as the key-value store & Flannel as the overlay network. I have also documented how to setup ‘Kube-dns’ at the end. I have created video tutorials on the same, for both Canonical & SLES. Although, my documentation specifically covers a setup of a single worker node. Following the documentation, it should be self-explanatory to set up more than one worker node. The final outputs of my work are as follows
- Setting up KVM on LinuxOne instance – https://gist.github.com/rajula96reddy/a9065c75d0b7c1b9f5472e0d480f31eb
- Analysing delay in Kubectl command – https://gist.github.com/rajula96reddy/dfdec14beac20859db4b47b61e79b7d2
- Final documentation hosted on GH pages
- Github repositories
- Video tutorials
So, this was my experience with the OMP Summer Internship Program. I know it was long, but I hope you had fun reading it. If you have any feedback, let me know in the comments below. Thanks for reading!
PS: If you are curious how to pronounce Kubernetes, it is pronounced as /koo-ber-nay’-tace/.
I would like to thank my mentor, Mike Friesenegger who was super nice & helpful throughout the program. I would also like to thank my professor, Prof. B. Thangaraju who introduced me to Kubernetes & helped me with the proposal. It was also nice working with my co-intern Asish Varanasi. I would also like to thank my Parents & friends who were supportive throughout the program.
Rajula Vineet Reddy