Category: Science Cloud

Who are the pilot users of the SNIC Science Cloud?

The SNIC Science Cloud is a project run by the Swedish National Infrastructure for Computing. The goal is to investigate if and how cloud resources should be provided as a complement to the more traditional HPC-resources. An important part of this investigation is our pilot users.  The project has run in two main phases:

1. 2013-2014: A small-scale pilot system and a small number (3-5) of predefined pilot use-cases, chosen to highlight the utility of cloud resources. Advanced user support in the form of SNIC Application Experts (AEs) were available to assist the pilot users.

2. 2015-2016: A scaled up infrastructure with more resources, and an open process for user-initiated project requests. In this phase, we have concentrated on training workshops and creating open source tutorials instead of targeted advanced support.

In the summer 2016 we reported on the growth of project requests and promised to follow up with a more in-depth analysis of who these users are and what they are doing. We have looked in our project database at all the projects that registered for SSC resources in 2015-2016 (as of Oct. 11). During the period, we have had 57 project requests.

As can be seen, Life Science users dominate. Of these, a large fraction are affiliated with SciLifeLab/NBIS. This is expected since the Bioinformatics community was an early adopter of service-oriented computing, and since their applications often have the need to integrate multiple software. Many Swedish universities are represented, but Uppsala University (UU), The Royal Institute of Technology (KTH) and the Karolinska Institute (KI) dominate. This is likely a consequence of the fact that these institutions’ involvement in SciLifeLab, and that both KTH and UU has served as hosts for the SSC project, presumably increasing awareness of SSC amongst the scientists.  

In the period Sep 11-Oct 11, a total of 47000 instance hours were deployed in 31 different projects. Using a reference instance type (flavor) with 4 VCPUs and 8GB RAM, this corresponds to an average of 130 instances continuously deployed during the month. We will follow up on the resource usage patterns over time in future posts, when we have more fine-grained data.

So what are the users doing with the SSC OpenStack resources? It appears that development and testing of software and services, as well as exploring the cloud computing paradigm for old and new types of applications are still the dominating use case. In common to most projects is the need for flexible customization of the computing environment, made possible by virtualization. Many projects also want to provide their solutions as services to serve their own specific community.

Some projects are making more substantial use of the IaaS resources, making use of advanced tools for contextualization, automation and orchestration to achieve quite a diverse range of objectives. In common to all these projects is that they have access to own expertise on distributed and cloud computing in the project groups. To serve as an inspiration to new users, we have during 2016 highlighted some of them as user success stories:

Elastic proteomics analysis in the Malmstroem Lab.

Processing ozone data from the Odin satellite at Chalmers University of Technology.

Estimation of failure probabilities with applications in underground porous media flows.  

Virtual Research Environment for Clinical Metabolomics.  

So what will happen in 2017? A projection is hard to provide, but given the global trend that private/community IaaS becomes more and more common also in academia, observations made by our partners in the Nordic Glenna project, and with the momentum created via the European OpenScienceCloud initiative, we believe that the interest in cloud resources will keep increasing rapidly. 

Fortunately, in the SSC project we are in a good position to meet an increased demand due to our architectural design based on regions, in which we can leverage previous generation HPC hardware at multiple geographic locations to quickly add compute hosts at low cost. We have now integrated resources at three HPC-centra, UPPMAX, C3SE and HPC2N and can if needed scale resources to over 5000 physical cores and 1PB of storage during 2017. This model also opens up for substantial user communities to enter SSC with their own dedicated regions. We also hope to start looking into public-private partnerships to secure a larger variety of SLA-backed resources and to allow for users to burst outside of the allocated quotas.

Scalable processing of ozone data from the Odin satellite

The Odin satellite has measured ozone and related gases in the stratosphere and mesosphere for more than 15 years. This is one of the longest data series with global coverage that exists in atmospheric science. To make sure we can provide the best possible reference data set to the science community ESA has funded a total overhaul of the data set. This includes everything from review of the calibration algorithm of the sensor, to a complete review of the algorithms that provide concentrations of the different species at different heights and locations in the atmosphere.

The instrument is a passive sub millimeter radiometer, measuring emitted energy from different molecules through the limb of the atmosphere. The atmosphere is scanned over different heights around 60 times an orbit and the satellite makes about 15 orbits a day. That way measurements quickly cover most of the Earth’s atmosphere. As an example we have been able follow the development of the ozone depletion over the polar regions. We can also follow the circulation of air masses globally.

Technology

The Odin science community is using the SNIC Science Cloud. This enables scalable processing of Odin data both for testing of alternative algorithms and standard products.

To be able to process the dataset as quickly as possible we have packaged a set of “processors” in docker images ready to be deployed at any docker enabled computer. These docker images are self-contained with auxiliary datasets and code ready to be fed with measurements from the Odin satellite. Once the docker images are deployed the container asks the Odin-API, a REST service, for the next available measurement and starts immediately to crunch the data. When the process is done it delivers the results back to the Odin-API and starts over with a new measurement.

Users can browse and download data from the user interface http://odin.rss.chalmers.se and power-users can communicate directly with the REST-API to analyse data programmatically or start new processing campaigns.

Important milestone reached for SNIC Science Cloud!

Our work on the SNIC Science Cloud is progressing at a steady rate, and we are very happy to announce that we have achieved an important milestone: over 50 active projects with more than 200 users from different disciplines. As the figure shows, the number of projects has been growing steadily, with an average of 3 per month.  Early in the project, we were driven by pre-defined pilot use-cases, but as can be seen, users are since some time ago finding us rather than the opposite. The question now is what will happen later in 2016 and 2017, will the linear trend continue or will we see a more rapid increase in project requests?

projects
Number of active projects on the SSC IaaS resources as registered in SUPR, the SNIC project management portal. The projects vary in size, and in total include over 200 users.

The IaaS is still based on best possible efforts and in kind contributions from three SNIC centers (UPPMAX, C3SE and HPC2N). We are currently working hard on implementing more extensive accounting, gathering data on resource usage patterns, and we will follow up with a more in-depth analysis later this year. This is an important activity in our efforts to better understand the operational costs and constraints of a cloud for science, and will form a basis for e.g. informed capacity planning. We plan to make all data publicly available in the hope that it will help other academic institutions in their e-infrastructure strategy work.

We have organized three workshops and there will be more to come in the fall. The aim of the workshops is to help the user community make productive use of clouds in their research.  The first workshop was held at the Department of Information Technology, Uppsala University (September 2015), the second was at SciLifeLab, Stockholm (March 2016) and recently the third workshop was held at Royal Institute of Technology (KTH), Stockholm (May 2016). The first two workshops were about introduction to Cloud computing and how to get started with OpenStack based IaaS. The third workshop address advanced concepts of virtualization, contextualization based on CloudInit and Ansible and orchestration using Heat.

Our senior cloud architect and UPPMAX application expert on cloud computing, Salman Toor, lecturing about cloud computing at KTH. In this advanced-level workshop, contextualization, orchestration and automation was on the agenda.
Our senior cloud architect and UPPMAX application expert on cloud computing, Salman Toor, lecturing about cloud computing at KTH. In this advanced-level workshop, contextualization, orchestration and automation was on the agenda.

SSC aims at being a very open project, sharing as much of our developed content and practices as open source. In line with this,  all the tutorial material is available for download, reuse, and importantly, contributions, from our GitHub repository:

https://github.com/SNICScienceCloud/technical-training.git

Pull requests are much appreciated.

Have a nice summer!

SSC Team

Using cloud computing for estimating failure probabilities with applications in underground porous media flows

In this guest post, Fredrik Hellman, a PhD student at the Division of Scientific Computing, Department of Information Technology, Uppsala University, report on how cloud computing resources in SSC were used in recent work with collaborators at UU and Chalmers/GU. 

In many engineering applications the probability of system failures are of particular interest. A special application is the assessment of storage capacity of underground carbon dioxoide storage reservoirs,where a failure is that the capacity of the target reservoir is smaller than expected. Since the rock properties are generally uncertain, the uncertainty in the reservoir capacity is also large.

The SNIC Science Cloud was used in our work on estimating failure probability to assess the performance of four different Monte Carlo method setups for estimating failure probability in a porous media fluid flow simulation with uncertain rock properties. For all four methods, the basic algorithm was to generate a set of realizations of the uncertain rock properties and distribute the work of performing the simulation for each realization in a network of virtual machines in the SNIC Science Cloud. All algorithms thus exhibit single program, multiple data (SPMD) parallelism.

snicblog

The code performing the simulations was written in Python, using
finite element assembly routines from the FEniCS project. The project benefited from using a cloud based service mainly for two reasons. First, the virtualization allowed for good control over the software environment. Experimental versions of software could easily be used without administrative overhead. Second, the IPython based MOLNs software for setting up and managing a virtual computing network for distributed computations was readily available and simplified the management of the computations.

Mini-workshop on microservice platforms

The Spjuth group is organizing a mini-workshop on microservice platforms on May 12 with talks 13:15-15:00 in ITC 2345, open to the public. SSC’s senior cloud architect Salman Toor will give a brief presentation about the cloud infrastructure. Agenda for the afternoon session is available here.

The type of microservice platforms presented offer higher-level functionality  by abstracting away most of the IaaS layer, allowing users to focus more on deploying applications than directly managing VMs. This makes it easier to build and deploy robust and scalable cloud computing applications. In the next SSC training session at KTH, there will be more information about MANTL, including a hands-on tutorial.

SNIC Science Cloud Workshop (May 2016)

Overview:

Instructor: Salman Toor.
Level: Basic.

Instructions: https://github.com/SNICScienceCloud/technical-training/tree/master/automation

Location: KTH, Stockholm.

Visiting address: Room 304, PDC Teknikringen 14, 3rd floor, KTH Stockholm

Infrastructure: SNIC Science Cloud (OpenStack based Community Cloud).

Date & duration: 31:st May, (10:00 – 16:00).

Audience: Users and potential users of SNIC Science Cloud resources with basic understanding of IaaS.

Registration:

Register here.

Topics:

  • Introduction to SNIC Science Cloud (SSC).
  • Virtual Machines and Containers.
  • Resource Contextualization.
  • Cloud Service Orchestration.
  • Mantl, A Framework for Microservices Infrastructure.

Hands-on session topics:

1 – API based communication
2 – Resource contextualization using cloud-init
3 – Examples of automation with ansible-playbook
4 – Heat Orchestration Template (HOT)
5 – Getting started with Containers

Lab-Document (not ready yet)

Schedule:

First half (10:15 – 12:00): Lectures
Second half (13:00 – 16:00): Lab session

 

Applied Cloud Computing Workshop (Spring 2016)

Overview:

Instructor: Salman Toor.
Level: Basic.

Location: SciLifeLab Gamma Level 6 (Pascal), Stockholm.

Visiting address: Science for Life Laboratory, Tomtebodavägen 23A, 17165 Solna, Sweden.

Infrastructure: SNIC Science Cloud (OpenStack based Community Cloud).

Date & duration: 30:th March, (10:00 – 16:00).

Audience: Users and potential users of SNIC Science Cloud resources with no previous cloud experience.

Registration:

Registration has closed.

Topics:

  • Brief overview of Cloud Computing.
  • Cloud offerings: Compute, Storage, Network as a Service (*aaS).
  • Brief description of IaaS, PaaS, SaaS etc.
  • How to access Cloud resources?
  • Introduction to SNIC Science Cloud initiative.

Hands-on session topics:

1 – How Horizon dashboard works?
2 – How to start a virtual machine (VM)?
3 – Instance snapshots.
4 – Access to cloud storage (volumes and Object store)
5 – Storage snapshot
5 – Network information
6 – Basic system interaction with APIs

Lab-Document

Schedule:

First half (10:15 – 12:00): Lectures
Second half (13:00 – 15:00): Lab session

 

Plans for first half of 2016

Part of the SNIC CLoud Team 2016
Part of the SNIC Cloud Team 2016. From left: Lars Viklund (HPC2N), Daniel Nilsson (C3SE), Andreas Hellander (UU), Salman Toor (UU, UPPMAX), Pontus Freyhult (UPPMAX) and Mathias Lindberg (C3SE). Missing from picture: Ingemar Fällman (HPC2N) and Henric Zazzi (PDC).

Last week we held out first all-hands meeting for 2016. Many of us were able to meet at HPC2N in Umeå for almost two days of brainstorming and technical work. Since we now have a functioning (but not yet production grade) IaaS cloud up and running, serving approximately 40 projects and 110 users, the focus of this meeting was on monitoring (increase stability), metering and accounting. We are, like all SNIC-supported projects, relying on the SUPR system for managing projects and users, but we haven’t yet developed a custom entry point for the cloud resources (we have been using “UPPMAX Small” templates, for those of you who know what that is). During the meeting, we completed a draft of the SUPR/SAMS workflows for cloud projects, in collaboration with representatives of the SAMS team. This is now to be handed off to those teams for feedback, and hopefully quick implementation.

Some other highlights from our all-hands meeting:

  • We decided to host 3 training workshops this semester targeted at beginning users for the cloud resources, tentatively at KI (end of February), Chalmers (late March) and Umeå University  (May). We will then follow this up with a more advanced workshop, showing some more advanced concepts and tools  in Uppsala early in the next semester.
  • We are in good shape to start accepting more users, now that we have two regions online. If you are interested, go ahead and make a project request.
  • We spent a lot of time discussing the incentive for users to make sensible use of the IaaS resources when developing applications. We will implement some form of pay-as-you-go model to promote dynamic use of resources. More information will follow.
  • We are planning to harden the systems, so as a user you will successively see a more and more stable system over the next couple of months. One step in that direction will be taken during the next large service window in the UPPMAX region Feb 15-Feb 29.
  • A third region at C3SE is well on its way.

SNIC Science Cloud – A Community Cloud and a Community Effort

We are happy to announce SNIC Science Cloud (SSC), a community cloud offering Infrastructure as a Service (IaaS), and, in the near future, selected Platform as a Service (PaaS) offerings free of charge to individual researchers at Swedish universities. We will start taking on more users during the next couple of months so let us know if you have a need for cloud computing infrastructure.

Open source. We are building SSC on the open source OpenStack cloud suite. Currently, we are hardening the system for sustained production. We are also scaling it to multiple regions with participation from the HPC centre UPPMAX (Uppsala), C3SE (Göteborg), HPC2N (Umeå) and PDC (Stockholm) to ensure that we can meet an increasing demand.

A community effort. Our goal is to provide a modern, flexible and open infrastructure that complements existing HPC resources. We strive for a community effort that evolves with and for researchers. We would love to hear from potential users regarding the needs for platform level services, such as Apache Hadoop/Spark, Kubernetes or other toolchains so that we focus efforts where they are most needed. What large datasets would you like to process?

Transparency to help others follow. In taking on the challenge of deploying and operating an OpenStack community cloud on a national scale over several hundreds of servers and many thousand physical cores, we hope to lead the way for other institutions that are considering similar initiatives. This is why we aim for transparency, both with architecture planning, operation practices (e.g. sharing code for testing and evaluation), and with data regarding usage patterns.

Open science, open data. With SSC we hope to take a leap towards an infrastructure for open science and open data, with cloud technology facilitating shareability and reproducibility of complex and computationally demanding experiments. We aim at making computations and data analysis more accessible for research communities with little previous experience of advanced and large scale computing resources. We are always interested in discussing these issues and in sharing and sharpening our vision.

You can help. Finally, there is a lot of work to do! If you are involved with academia in Sweden and you are an OpenStack operator, have experience of e.g. software stacks for large scale data processing, microservices orchestration, automation, or if you belong to a community that is using some specific SaaS that you would like to provide for research groups, we want your help!