The Big Data Strategy

 

Strategy. 

For data analytics, T-Systems differentiates by volume: small (GB), big (TB), huge (PB).

Due to the multi-platform strategy pursued by T-Systems, we can offer solutions in all the three volume categories, ranging from private, public onto dedicated, dynamic cloud platforms.

The primary platforms/products offered by T-Systems are: Cloudera Hadoop, Talend, Tableau and Microsoft Azure.

The broad strategy is outlined below.

bigdata-strategy

Vendor-neutral

T-Systems is vendor-neutral, and strives to offer the most appropriate solution for its customers.

We start with analyzing the customer situation, making a gap analysis for the future scenario, analyzing a couple of possible solutions, and recommending the best fit.

Customer Story

Let us assume that the customer comes from the Manufacturing segment, and wants to analyze machine data in a factory premise. First, we prepare a customer story, that describes the customer situation, the goal, and the means to achieve it.

A Manufacturing Customer Story could look like this:

customer-story_manufacturing

For demo purposes, we often use Microsoft Azure with PowerBI for dashboard features.

Sample dashboards for Manufacturing customer are shown below, prepared in PowerBI, with data analysis in Microsoft Azure.

englishdashboard

 

 manu2

 

 

Process

The principal process chain behind our approach is as follows:

INGEST > PREPARE > STORE > ANALYZE > OUTPUT

Ingest: this phase collects the data from the physical world into the analytics platform. The physical world comprises field sensors and machine parts, but also social media like Twitter updates. This process also comes out to be the most complex, as integration of varied kinds of sensors adds complexity to the system.

Prepare: this phase cleanses the data, identifies gaps, gives structure, and prepares it for storage in the data platform.

Store: this phase actually stores the data. Two primary modes are supported: hot (real-time) and cold (for deeper, more complex analytics in a longer time period).

Analyze: this phase is the core analytics engine, where much of the data processing takes place. Insights from data are generated in this phase. This is normally the most expensive component in the analytics chain.

Output: this phase is the customer interface, thus the most visible part of the analytics chain. Typical output modes are computer screen and printed reports, but of late, tablets and mobile phone notification have gained immense importance.

 

Pricing

In a typical T-Systems offering, there are 3 price components, as shown below.

pricing

Product Architecture is a one-time activity, at the beginning of the project, and is charged as a fixed one-time fee.

The System Implementation is charged by man-days, and can be compared to a consulting offering.

The Managed Hosting is a running cost, charged per month, for a managed cloud environement, hosted and managed end-to-end by T-Systems for the customer.

 

CI/CD: Jenkins 2 on openshift

Jenkins 2 overview and setup – APPAGILE

We specifically unveil the steps with oc and an example workflow for Jenkins 2.

IMPORTANT: after installing the OpenShift plugin for Jenkins, always check the box enabling the OpenShift builder support (also for the related OpenShift Jenkins builder API) in the build configuration panel of your project.

Setup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# setup ci jenkins
oc new-project ci
oc new-app library/jenkins:2.0
oc expose svc jenkins
# setup sample for jenkins build
oc new-project demo
oc new-app eap64-basic-s2i \
--param=APPLICATION_NAME=demo-jenkins \
--param=SOURCE_REPOSITORY_URL=<HTTPS>://gitlabappadev.tsi-af.de/julien.siebenthal/demo-jenkins.git \
--param=SOURCE_REPOSITORY_REF=2.7.0.Final \
--param=CONTEXT_DIR=demo
# setup a secret to access correctly the private repos if needed
oc secrets new-basicauth basicsecret --username=<your_username> --password=<your_password>
oc setbuild-secret --sourcebc/demo-jenkinsbasicsecret
# setup policies
oc policy add-role-to-user edit system:serviceaccount:ci:default -n ci
oc policy add-role-to-user edit system:serviceaccount:demo:default -n demo
oc policy add-role-to-user edit system:serviceaccount:ci:default -n demo

On the Jenkins pipeline side here is the Groovy script based on the Ticket-Monster example.

We give the export of the buildconfig that can be used in relation with oc create :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: v1
kind: BuildConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: null
  labels:
    app: eap64-basic-s2i
    application: demo-jenkins
    template: eap64-basic-s2i
    xpaas: 1.3.2
  name: demo-jenkins
spec:
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: demo-jenkins:latest
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    contextDir: demo
    git:
      ref: 2.7.0.Final
      uri: <HTTPS>://gitlabappadev.tsi-af.de/julien.siebenthal/demo-jenkins.git
    sourceSecret:
      name: basicsecret
    type: Git
  strategy:
    sourceStrategy:
      forcePull: true
      from:
        kind: ImageStreamTag
        name: jboss-eap64-openshift:1.4
        namespace: openshift
    type: Source

Security

To pull/push from/to a private repo, setup with the Credential binding plugin a new domain and credential, see Jenkins access to Gitdev private repo using ssh, to Gitlabappadev using https

  • user/private_key for ssh based pull/push
  • user/password for https based pull/push

use a git https based approach use (gitlabappadev.tsi-af.de, seems git ssh not enabled) :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
node {
  stage 'Checkout'
  git branch: '2.7.0.Final', credentialsId: '3a6a12be-0b9a-45af-9c41-4f21fa1543e3', url: '<HTTPS>://gitlabappadev.tsi-af.de/julien.siebenthal/demo-jenkins.git'
  // ** NOTE: This 'M3' maven tool must be configured in the global configuration.
  defmvnHome = tool 'M3'
  stage 'Build'
  sh "${mvnHome}/bin/mvn -f demo/pom.xml clean install"
  stage 'Deploy'
  defbuilder = newcom.openshift.jenkins.plugins.pipeline.OpenShiftBuilder("""demo-jenkins""demo"null"""""""""true""""""")
  stepbuilder
}

Jenkins web console

From there you can directly interact with the web console of your project.

screen%20shot%202016-11-24%20at%2009_38_12

Jenkins access to Gitdev private repo using ssh, to Gitlabappadev using https:

  1. Access your jenkins pod
    1. access jump server
    2. locate where your Jenkins pod run (which node), then ssh to the corresponding machine
    3. from there in the shell:
      >> docker exec -it <jenkins_pod_ID> /bin/bash
    4. cd in the pod session
    5. you should be in the /var/jenkins_home
    6. create a ssh key in my case:
      >> ssh-keygen -t rsa -C “USERNAME@masterd3.tsi-af.de
    7. it asks for a passphrase, you should put one and record it somewhere safe
    8. a pair is created, by default the id_rsa (private key) and id_rsa.pub (public key)
    9. not finished, once done you must be sure you can access the gitdev (gitlab) server, type :
      >> ssh -T git@gitdev.tsi-af.de
    10. during the authentication process the daemon asks for the passphrase you have introduced earlier if you set one, give it
    11. the ssh daemon will ask you if you want to authenticate to the gitdev server, obviously yes.
  2. In the Jenkins web console
    1. be sure to have the proper module for ssh authentication so to say:
      1. SSH-Agent plugin
      2. SSH plugin (normally it is required by the first one and will be installed automatically)
      3. Credentials binding plugin
    2. restart the Jenkins server once the plugins are downloaded
    3. then you should go into the Credentials option displayed from now on the web console
      1. you should define a domain
        1. set this to the hostname gitdev.tsi-af.de
      2. and from this domain a key based new credential
        1. indicate you want a ssh authentication with private key and indicate the correct location of it (in my case /var/jenkins_home/.ssh/id_rsa) and do not forget to set the passphrase
        2. IMPORTANT: leave the credentialID field empty, once you save, one will be created automatically for you. This ID is tremendously important, it will be added into the Jenkinsfile workflow script
        3. save the credential.screen%20shot%202016-11-18%20at%2012_03_27
    4. go in your workflow build project and adapt the script, mine is:
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      node {
        stage 'Checkout'
        git branch: '2.7.0.Final', credentialsId: '9cae6b3a-4437-4a33-b99e-c3174f90670f', url: 'git@gitdev.tsi-af.de:jdesiebe/myticket-monster.git'
        // ** NOTE: This 'M3' maven tool must be configured in the global configuration.
        def mvnHome = tool 'M3'
        stage 'Build'
        sh "${mvnHome}/bin/mvn -f demo/pom.xml clean install"
        stage 'Deploy'
        def builder = new com.openshift.jenkins.plugins.pipeline.OpenShiftBuilder("""ticket-monster""demo"null"""""""""true""""""")
        step builder
      }
    5. As you see in the script, a credentialID field is added and refer to the credential we added just before, also we indicate to Jenkins that we want to use the ssh prototcol to pull the git repo.
  3. Restart your build, it should work (Lächeln).
  4. if you want to get momentarily access to private repos from gitlabappadev.tsi-af.de use https
    1. in that case set a new domain and a credential with username/password setup, no need for a key

Jenkins based documentation on the web related to workflows and security

Creating Docker from public repository

When you use the web-console from OSE/AppAgile, you find only images which are provided by AppAgile locally:

bildschirmfoto-2016-12-09-um-14-27-21

But if you like to test an external image, you cant use the docker-hub URL.

To do so – use the command line interface:

Login to your environment (./oc login xxx-podname-xxx)

Then create a new app from command-line (phpMyAdmin in that example):

MacKeks:ose szosel$ ./oc new-app phpmyadmin/phpmyadmin

--> Found Docker image 41e518d (3 days old) from Docker Hub for "phpmyadmin/phpmyadmin"
 * An image stream will be created as "phpmyadmin:latest" that will track this image
 * This image will be deployed in deployment config "phpmyadmin"
 * [WARNING] Image "phpmyadmin" runs as the 'root' user which may not be permitted by your cluster administrator
 * Port 80/tcp will be load balanced by service "phpmyadmin"
--> Creating resources with label app=phpmyadmin ...
 ImageStream "phpmyadmin" created
 DeploymentConfig "phpmyadmin" created
 Service "phpmyadmin" created
--> Success
 Run 'oc status' to view your app.
MacKeks:ose szosel$

Voila – go back to the web client and check, how that image is being deployed.

Note. you see the name phpmyadmin/phpmyadmin.

On DockerHub the images are sorted in different folders – if you find the right image on DokerHub – just copy that folde/name combination. The oc-command line will search automatically on the public ressources and installs the image.

From oc-help:

MacKeks:ose szosel$ ./oc new-app -h
Create a new application by specifying source code, templates, and/or images

This command will try to build up the components of an application using images, templates,
or code that has a public repository. It will lookup the images on the local Docker installation
(if available), a Docker registry, an integrated image stream, or stored templates.

If you specify a source code URL, it will set up a build that takes your source code and converts
it into an image that can run inside of a pod. Local source must be in a git repository that has a
remote repository that the server can see. The images will be deployed via a deployment
configuration, and a service will be connected to the first public port of the app. You may either specify
components using the various existing flags or let new-app autodetect what kind of components
you have provided.

If you provide source code, a new build will be automatically triggered.
You can use 'oc status' to check the progress.