Introduction to the Video Intelligence API

Posted on in GCP Machine Learning

Search and discover your media content with powerful Cloud Video Intelligence API

Google Cloud Video Intelligence API makes videos searchable, and discoverable, by extracting metadata with an easy to use REST API. You can now search every moment of every video file in your catalog and find every occurrence as well as its significance. It quickly annotates videos stored in Google Cloud Storage, and helps you identify key nouns entities of your video, and when they occur within the video. Separate signal from noise, by retrieving relevant information at the video, shot or per frame.

You can try this out, yourself:

  1. Open: https://cloud.google.com/video-intelligence/#demo
  2. Select a sample video from the list, for example: Volleyball Court, (which is a video made at the Google Mountain View office). Notice the labels.
  3. Select another sample video from the list: Google Work.
  4. Click on the Shots tab. Notice all the keywords detected from the video, which are being renewed per video shot!
  5. Click on the API tab. Notice how the JSON response would look like.
  6. Now, lets try it with one of my own videos, which I’ve uploaded as a public available video in Cloud Storage: gs://leeboonstra-videos/mov_bbb.mp4

Write metadata on video upload

Machine Learning for videos, can be super interesting, in case you want to implement it within your own systems. Let’s say you host a lot of videos on your website. Instead of manually writing meta per video; you could create an ETL job, (for example through Cloud Functions), which listens to the upload event of Google Cloud Storage, runs the Video Intelligence API, and writes the metadata in a database.

This looks like a great use case! Let’s see if we can build this!

Getting Started

To get started, open the GCP console, and create a new project. Go to: https://console.cloud.google.com and click: Create new project, and enter an easy to remember project name, which will be used as the project id. You will have to remember this.

The project id will have to be unique. For this workshop, it might be a good practice to prefix it with your [firstname][lastname]

Enable the Cloud Functions API

Click on the menu button. Cloud Functions > Enable API.

Please mind, the Cloud Functions API is in beta.

Enable the Video Intelligence API

Click on the menu button. API Manager. Search for: Google Cloud Video Intelligence. API > Enable

Click Credentials > Create Credentials > Service Account

Choose New Service Account

Service Account Name: demoapp

Role: Project Owner

Choose JSON

Click Create

Click Manage Service Accounts

Click the menu button (the 3 vertical dots), and select Create Key

This will download the key on your local machine. Create a folder on your machine called: cloudfunctions-videoapi, and move the file over.

Create storage buckets

When you write the JavaScript code for the cloud function, you will need to upload it somewhere. Under the hood, GCP will create a container with a Node environment for you, so you can execute your function. You can upload function code, in a bucket of the Google Storage. Let’s create a bucket which contains function code.

You can create the bucket via the Cloud Console (menu > Storage > Create Bucket), or from the command-line, on your local machine (if you have the gcloud sdk installed), or from your online terminal in the Cloud Console:

gsutil mb -p [PROJECT_ID] gs://[BUCKET_NAME]

Create two buckets, with the name of your [project id] + -functions-src, and one [project-id]-videos. (This way, your bucket will be unique.)

After you’ve created the videos bucket, you will need to make this bucket public, so your video’s get gs:// public urls. You can do this by clicking on the menu button of the bucket (the button with the 3 vertical dots). Click Add Item:

User - allAuthenticatedUsers - Reader

Click Save.

Client Library

Since the client node js library is still in Alpha at the time of writing, we will download the alpha version and host it locally. Once the library gets publically available, you can ignore the next step, and instead link to the online version of the client library. (See: https://www.npmjs.com/package/google-cloud)

Download the alpha node client library:

https://storage.googleapis.com/videointelligence-alpha/videointelligence-nodejs.tar.gz

Unzip the package. Create a new folder inside cloudfunctions-video, called: local_modules, and move the videointelligence folder into it.

Create a cloudfunctions-video/package.json file. It should contain the following contents:

{
    "name": "videoapi",
    "description": "Webservice VideoAPI",
    "version": "0.0.1",
    "private": true,
    "license": "Apache-2.0",
    "author": "Lee Boonstra",
    "engines": {
        "node": "~4.2"
    },
    "dependencies": {
"@google-cloud/videointelligence": "file:local_modules/videointelligence",
        "dotenv": "^4.0.0"
    }
}

Create a Cloud Function

Create a new hidden .env file, and give it the following content:

ENV=production
GCLOUD_PROJECT=[project-id]
GCLOUD_KEY_FILE=./[path to json service key file]

Afterwards, create the index.js file with these contents:

Deploy your Cloud Function

On the command-line, enter the following:

$ gcloud beta functions deploy [name-function] --stage-bucket [bucket name of the function-src bucket] --trigger-bucket [bucket name of the videos bucket]

For example:

$ gcloud beta functions deploy videoapi --stage-bucket leeboonstra-functions-src --trigger-bucket leeboonstra-videoapi

TimeOuts

The way how the video api works, is that it first will read in the video to the memory. The machine learning under the hood, is similar to Google’s Cloud Vision API, which can detect what’s on an image. But for the Video Intelligence API this works per frame.
Cloud functions can timeout. You will need to specify a timeout in seconds. By default it’s set to 60s. A number between 1 and 540 (9min) is required. A video with a long duration, will likely make that the cloud function will timeout. So becareful.
You can either setup, the timeout in the Cloud Functions / <myfunction / EDIT menu. Or pass it in directly from the command-line:

$ gcloud beta functions deploy videoapi --stage-bucket leeboonstra-functions-src --trigger-bucket leeboonstra-videoapi --timeout=540

Testing the function

Go to Storage, and click on your videos bucket.
Click Upload Files, upload a mp4 file (such as: http://www.divx.com/en/devices/profiles/video ) to the bucket. You will need to make the video public available. Once you’ve done, it will give you a public link.

Once, it’s uploaded, we can check the logs, to see if all the tags are logged. Every time you use console.log() in your Cloud Function code, you will write data to your logs. You can access these logs, by opening your Cloud Console > Logging

Conclusion

By now, you managed to test the Video Intelligence API with your own JavaScript code, within a cloud function based on file upload in a bucket. The power of this cloud function, is that you could easily build an application around this, which makes use of microservices (cloud functions).

It wouldn’t be much work to create an interface (for example with Sencha Ext JS, or just with plain HTML, CSS and JavaScript), which shows a list of video’s and the tags.

I’m logging the tags in the logs of Stack Driver. But instead I could save it in the data store. I could create another cloud function, based on an HTTP trigger which loads the data of the datastore and displays it in the front-end list.

Another idea could be, to pass in the results of the Video Intelligence API into another Machine Learning API, such a translate. To translate the keywords to a different language, before saving it in the database.

TIP: In case you don’t want English meta data, it’s also possible to put the Translate API right after the Video Intelligence API call!

Video Intelligence API from Ladysign on Vimeo.

Resources


* Cloud Functions Docs
* Video Intelligence Demo

Analyzing data with BigQuery and Machine Learning APIs

Posted on in Data science GCP Machine Learning

Forecast the Dutch elections based on social media presence.

Wednesday the 15th of March 2017, was a big day for the Netherlands, at that day it was the Dutch general election, where the Dutch elected members of the house of representatives. The House of Representatives (the Second Chamber) is composed of 150 seats elected by proportional representation in a single nationwide constituency. There are 28 political parties, where the Dutch can vote the members from.
Since I recently started working with Google Cloud Platform, I thought it would be a great idea, to predict the winning political party based on social media presence and sentiment. This idea is based on the blog post of Sara Robinson: Comparing tweets about Trump & Hillary with natural language processing.

This blog post will contain the following technical pieces:

  • Compute Engine (1) - To deploy our data scraping script on a VM.
  • Machine Learning APIs - To use Natural Language API (4) to understand the context of the data, and since our data is in Dutch (and the Natural Language API doesn’t support the Dutch language yet), we will need the Translate API (2) to translate.
  • BigQuery (5) - To collect a lot of data. To analyze this data we use BigQuery and run some queries on it.
  • To visualize our result set, we will make use of Google’s Data Studio (6). We can use some nice charts!

To do the social media data scraping, we use the Twitter Streaming API. The application code is written in JavaScript for Node.js.

Let’s get started with this tutorial, and see if we can predict which Dutch political party will win the elections!

Application Architecture with the various Google Cloud solutions

NOTE: This blog post won’t contain any political opinions from me, neither from Google. This is a blog post to demonstrate the various solutions of the Google Cloud Platform, on how you can be creative with data. Please see the disclaimer, at the end of this article.

Prerequisites

*Node JS
* Node Express framework ( $ npm install express --save)
* A GCP Account (or create a free trial account)

Getting Started

To get started, open the GCP console, and create a new project. Go to: https://console.cloud.google.com and click: Create new project, and enter an easy to remember project name, which will be used as the project id. You will have to remember this.

Next, on your local machine, generate an empty Node Express application:

$ express

NOTE: Technically, for this demo, you don’t need Express, since Express is a Web framework for Node.js. I choose it though, since I prefer working from this folder structure. My code will be future proof, should I ever plan to port it to a web app.

Once, the generation is done, you can open the package.json file, and add the following lines of code dependencies:

  "dotenv": "~4.0.0",
  "google-cloud": "~0.47.0",
  "user-stream": "~0.0.8",
  "nodemon": "~1.11.0",

After, you are done; run the following command in your terminal:

$ npm install

Go back to the GCP console and click the hamburger menu icon, and select: IAM > Service Accounts > Create Service Account

Service account name: nodeapp
Role: Project > Owner

Click: Create > Close

Click the button with the 3 dots next to nodeapp, to open an extra menu. Select: Create key. Choose JSON. Click Create.

A key has been saved to your local machine. Rename this -.json file, as: cloudkey.json, and copy it into your freshly generated Express app, root folder.

Also, create an .env file in the root of your Express app folder. It should have the following contents:

GCLOUD_PROJECT=<my cloud project id>
GCLOUD_KEY_FILE=./cloudkey.json

Later in this tutorial, you will also need to add the keys for the Twitter Streaming API in here.

Twitter Streaming API

The Twitter Streaming APIs give developers low latency access to Twitter’s global stream of Tweet data. A streaming client that pushes tweets, without any of the overhead associated with polling a REST endpoint. Exactly, what we need, to monitor the Twitter newsfeed during election day, so we can see which party is popular on Twitter.

We will need to create a Twitter API account. (and if you don’t have Twitter, also a Twitter account).

With the Twitter API account, you can create an application, and generate the tokens. These tokens, you will need to copy to the .env file in the root of your Node project. It has the following contents:

CONSUMER_KEY=<my consumer key>
CONSUMER_SECRET=<my consumer secret>
ACCESS_TOKEN_KEY=<my access token>
ACCESS_TOKEN_SECRET=<my access token secret>

Create a folder called: lib

Create the following file in the lib folder: twitter.js
The twitter.js file will contain the following content:

Now you can start testing your twitter stream. In your terminal run the following command from your Express root folder:

$ node bin/www

Now, look in your terminal, you should see the data based on your search terms streaming in. To close the data stream hit CTRL + c.

Machine Learning APIs

Nice all that data, but how will we know that people are talking good or bad about a certain political party? Just counting all the tweets, wouldn’t be the correct thing to do, we actually need to understand the context of the data. Hmm, how can we automate this?
For this, you can actually use machine learning. Machine learning is not only for data scientists and experts, you and I can make use of it as well.
The Google Cloud platform provides a bunch of APIs which are easy to use. Think about machine learning APIs such as: Speech API (which can convert speech to text), Vision API (which can get insights from pictures), Translate API (to translate text to a certain language), Video Intelligence API (to retrieve metadata from videos) and Natural Language API (to derive insights from unstructured text).
Google is planning to release much more machine learning APIs to the cloud platform, and on top of that; if you are a data scientists you could write your own machine learning models.

The last mentioned API is what we need to understand pieces of our Twitter data.
However... Unfortunately, GCP won’t understand the Dutch language yet. So here’s the test case: Would it work if we translate the text from Dutch to English first? Afterall, the only thing we need to know is if the context of the tweet is positive or negative. Let’s give it a try. We make use of the Translate API first, afterwards the Natural Language API. Later, we will store in BigQuery the original tweet, but with a ranking total score column.

Go to the cloud console and click on the menu (hamburger) button. Click API Manager > Dashboard and click Enable APIs. Click Natural Language API from the Google Cloud Machine Learning API section. Click Enable. Go back to the previous screen, and select Translation API and hit Enable again.

In the lib folder, create the following file: ml.js which contains these contents: (see the comments in the JavaScript code for further explanation).

In twitter.js, add the following line somewhere in the top of the file:

var path = require('path');
var machinelearning = require( path.resolve( __dirname, "ml.js" ) );

You can test the MachineLearning APIs by adding the following example code lines to the end of the file:

getSentiment('I love Milkshakes', function(results){
    console.log(results.sentences);
});

getTranslation('Ik houd van Milkshakes.', function(results){
    console.log(results);
});

The sentiment score is a number between -1.0 (negative sentiment) and 1.0 (positive sentiment). The magnitude is a non-negative number in the [0, +inf) range, which represents the absolute magnitude of sentiment regardless of score (positive or negative).

To change the language, change the from input parameter on line 11, to another language, such as es for Spanish, or fr for French.

Feel free to play arround with it, and pass in different text strings, or language API settings. From the terminal run:

$ node bin/www

Now, you will have to add these pieces of code, within the stream.on() function, right after the console.log line. (let’s say line 42). This will be a callback, in a callback, in a callback... pffff. ...and we also need to bind to the original scope! So I solved it this way:

Once done with it, start the node app again. You should see the twitter lines, streaming in, with translations and sentiment detection.

The next step will be to push these data into BigQuery!

BigQuery

The next challenge is to store this data in the BigQuery storage in the Google cloud.
BigQuery is a fast, economical and fully-managed enterprise data warehouse for large-scale data analytics. BigQuery can query terabytes of data in seconds and petabytes in minutes!

The code to save the data in BigQuery is not so difficult. Please have a look:

In twitter.js you will need to require the file:

var bigquery = require( path.resolve( __dirname, "bigQuery.js" ) );

And within the most inner callback (on line 127), you can enable the insertBq() method:

bigquery.insertInBq(row);

Run the node app for a while. This will load new content into BigQuery. We can now run a bunch of queries to analyze our data. Click Compose Query, write:

SELECT SUM(score) as totalscore,
  party,
  COUNT(party) as totalparties
FROM dutchelectionstweets.dutchelections
GROUP BY party
ORDER BY totalscore DESC

This query, will sum the total sentiment score per party. Also, it counts and it groups the parties.

Click Run, to run the query. You will see the results in the table below.
When the results look ok to you, you can save the view. This will create a new view (which is like a table, but it includes the query). It contains the unique result set. We will need this later for Data Studio.

Let’s create a few more views. Such as the 3 most positive parties:

SELECT SUM(score) as totalscore,
  party
FROM dutchelectionstweets.dutchelections
GROUP BY party
ORDER BY totalscore DESC
LIMIT 3

The total amount of tweets per party:

SELECT party,
  COUNT(party) as totalparties
FROM dutchelectionstweets.dutchelections
GROUP BY party
ORDER BY totalparties DESC

And the total amount of tweets:

SELECT COUNT(text) as totaltweets
FROM dutchelectionstweets.dutchelections

In case you want to browse through all the tweets you can use:

SELECT text, party, score
FROM dutchelectionstweets.dutchelections

Data Studio

Instead of displaying the results in "boring" tables, we could very easily display the result sets in charts. I’ve used Google Data Studio for this.

When you drop a chart on the stage, you will need to Create a new Data Source.
Which you can set to BigQuery Connector. From there, you select your project, your dataset, and then your table set or table views. From that moment, you can start using all your data within the Studio. You can easily refresh the data, by clicking on the Refresh Data button:

Refresh data in Data Studio

With Google Data Studio you can drag and drop charts, images and visualizations on the stage. It turns your data into informative dashboards and reports which are easy to read and share. Google Data Studio is currently in beta. It’s not part of Google Cloud Platform, it’s a Google Analytics solution. You can read more about this tool on: [https://www.google.com/analytics/data-studio/]

Compute Engine / Cloud Launcher

Ok, so we’re done with writing the code to fetch and analyze our data! Although you could keep your terminal open on your local machine for a full day, it’s not ideal. At the end, this process will take some processing power. A better use case would be to move it to a virtual machine.

Let’s create a virtual machine. Open the Cloud console in your browser.
You could create a virtual machine, manually yourself by clicking the Compute Engine menu option. But since we need an image which requires Node.js installed, we will do it the easy way, instead make use of Cloud Launcher!

  1. Click [https://console.cloud.google.com/launcher/]
  2. Search for Node.js
  3. Choose the 2nd Node.js image, (the one from Bitnami.)
  4. Click the Launch on Compute Engine button.
  5. Specify a name, and a zone (I am in Europe, so I choose europe-west-1b)
  6. Choose micro machine type.
  7. Hit the Deploy button.

Notice, how fast it will launch this virtual machine for you!

After you deployed the Bitnami Node.js app, you will see the IP address where your VM will be available. There’s an SSH dropdown. Hit it, and click View SSH command. It will show you the command that you can paste in your local terminal. For example:

gcloud compute --project "myproject" ssh --zone "europe-west1-b" "nodejs-1-vm"

This will log you in on the virtual machine. When you run this command the first time, you will need to create a passphrase. It will automatically create a public key on your machine.

Once you’re logged in, you can verify if nodejs was properly installed. Enter:

$ node -v

It should output the nodejs version number.

The next step is to transfer our project on this virtual machine.
You can use Github for this, or you can use Google Cloud Development repositories.
(I choose Github, since I want to share my public Github repository, with you. When you have your own project running locally on your machine, you can follow the steps from the Development screen instead.)

Pull the code, from some repository in the root of your vm:

$ git clone https://github.com/savelee/gcloud-bq-election-demo.git

Navigate into the folder:

$ cd gcloud-bq-election-demo/

Install all the nodejs packages:

$ npm install

We aren’t done yet! We need to have all our keys available on the virtual machine. That’s the .env file and the cloudkeys.json file. Obviously, I didn’t put these files in my public Github repo. Hmm, so how can we move these files to the VM? With AppEngine or ContainerEngine, you probably would work with Docker images, so you can easily put these keys in the Docker file. For this demo, we use Compute Engine, so we should make use of Google Cloud Storage instead. We can create a bucket, and upload our keys into it.

Open the Cloud console in your browser. Click on Storage > Create Bucket.
Give the bucket a name. (For example -contents>). Make sure this bucketname is unique, and remember it, because you will use it later.

Select Regional, and select the same region as the VM you have choosen before. (so europe-west1 in my case.)
Click Create.

Once the bucket is created, we upload the files from the command-line. In your terminal on your local development machine enter the following two commands:

$ gsutil cp .env gs://<name-of-my-bucket>
$ gsutil cp cloudkey.json gs://<name-of-my-bucket>

Both files are now in the bucket. The next step will be to download it, from the bucket into your virtual machine.

Open the terminal which runs on your VM. Enter the following command, to download everything from the bucket, into the root of your vm:

$ gsutil cp gs://<name-of-my-bucket>/* .

If you want, you can remove your bucket in the console. (Although, it probably wouldn’t cost much, to save 2 small files.)

And now you can start the Node.js service:

$ nodemon bin/www

And now you can start the Node.js service.
You might have noticed, that when you start running nodemon, and you close the VM console, that your script stops as well. What you will need is a Node module, which can run your script “forever”.

$ npm install forever -g

And then start your script like:

$ sudo /opt/bitnami/nodejs/bin/forever start bin/www

(with sudo forever stopall you can stop the script again)

You should see the Tweets coming by. When you query in BigQuery you will see the results! Awesome, now let’s keep this process running for a while!

TIP: In case you rather want to schedule your script to go on, on a certain time,
you could use Cron for this.

First create a symlink to the correct node folder:
$ sudo ln -s /opt/bitnami/nodejs /usr/bin/node

Then start a crontab

$ crontab -e

Inside the crontab you code probably would look something like:

SHELL=/bin/shell
0 0 15 3 * PATH=$PATH:/opt/bitnami/nodejs cd ~/gcloud-bq-election-demo/ && sudo /opt/bitnami/nodejs/bin/forever start bin/www

This means, at 0:00:00 time, on the 15th of March, navigate to the gcloud-bq-election-demo directory, and start the nodemon script. You can request the
VM time by running date in the console.

You can check if it all worked out, by viewing the logs:

$ tail -f /var/log/syslog

Conclusion

By now, you have learned how to work with various Google Cloud services. We have been looking into: Compute Engine and Cloud Launcher to spin off a VM in the cloud, Machine Learning APIs like the Natural Language API and Translation API to understand the data, BigQuery to analyze the data and Data Studio to visualize the data in nice charts.

Did I predict the Dutch elections well? Well, in some extend. The top 2 most positively spoken parties on Twitter, got the 2nd and 3rd place in the final results. So that was as expected. The PvdA didn’t do well at all, and the VVD got the first place in the elections, with the most votes. If I do random checks on the collected data, I can see that the sentiment detection has been calculated correctly. There is just no real direct relation between popularity on social media vs. the voting process. But mind you every person can only vote once, but can post unlimited positive or negative tweets.

All my code will be in Github [https://github.com/savelee/gcloud-bq-election-demo]. Feel free to use my code, to run a similar case. I will put the instructions on how to setup, in the readme file.

In case you are new to Google Cloud platform. You can create a free trial account, which includes $300 dollars that can be used to spend on cloud services which expire after 12 months. That should be enough money to try out this demo.

The costs

To run this tutorial for one day in a Compute VM created by Cloud Launcher, with the use of machine learning APIs and streaming and querying within BigQuery, costed me about 14 dollars. This included the VM running for 5 days (which costed me a dollar). My data wasn’t that big, so I didn’t had any costs for executing BigQuery queries. I was worried about the amount of API calls that I would made, but by running the script for a large part of the election day, I managed to still stay mostly within the free Natural Language API call request tier. So most of my money has been used on the large amount of tweets that has been translated.

Let’s drill a little more into the costs of GCP.
BigQuery storage will cost, $0.02 per GB Streaming data in BigQuery: $0.05 per GB (loading data is free). To execute Queries, will cost you $5.00 per TB but the first TB per month is free. BigQuery Pricing. Note, that Data Studio makes use of a BigQuery connector, which basically fires queries in BigQuery.

The price for a micro virtual machine will cost less than 5 dollar a month. But for a computing demo like this, you will probably only run it for a bit, and pay for your use. With Compute Engine, you will pay per minute, with a 10 minute minimum.

The prices for translation and language detection is about 20 dollars for a million characters. It will become cheaper, once your amount is over the 1.5 billion characters.
See Translate API pricing.
The Natural Language API has a free tier, if the amount of text records stays under the 5k, for pricing info see: Natural Language Pricing.
Another great way, for checking and estimating costs, is by making use of the GCP Pricing calculators.

In case you are concerned about the prices, you can setup a billing alert. Which can send an email to you once you reach your quota. From the Cloud console, you can enable this, by clicking: Billing > Budgets & Alerts.
In this screen, you can select the project, and your quota. - You can get emails once you get closer to your quota.

Disclaimer

This blog post won’t contain any political opinions from me, neither from Google. This is a blog post to demonstrate the various solutions of the Google Cloud Platform, on how you can be creative with data.

I have collected a large part of the day all the Dutch political tweets, posted on Twitter. Though, my results can not be considered as the ultimate truth.

I use the Natural Language API in combination with the Translation API, translating first from Dutch to English before doing a sentiment analysis. Whether this always results, in the right score is the question. But based on some random tests, the majority of the tweets seem to have a reasonable score. At the end I only need to figure out if the score is positive or negative. Let’s look into an example:

Here’s the translation of the translate.translate() call with the following text: Partij XYZ is mijn favoriete partij. Met de meeste stellingen ben ik het eens..
It will be translated to this: Party XYZ is my favorite party. With most positions, I agree. If I would have translated it myself, I probably would have said it a little different: "Party XYZ is my favorite party. I agree with most of the positions they take on issues." However, to detect the sentiment, it doesn’t really matter, the score seems to be positive and correct:
{ magnitude: 0.800000011920929, score: 0.800000011920929 }

There are certain tweets which I left out the totalscore calculation for sentiment analysis. For example, tweets which contain multiple opinions for multiple political parties in one single tweet:

Here’s the result of the language.detectSentiment() call with the following text: I love Milkshakes. :
{ magnitude: 0.699999988079071, score: 0.699999988079071 }

Here’s the result of the language.detectSentiment() call with the following text: I really hate Whipcream. :
{ magnitude: 0.5, score: -0.5 }

Here’s the result for the language.detectSentiment() call with the following text: I love Milkshakes but I really hate Whipcream. :
{ magnitude: 0, score: 0 }

I’m sure there are ways on how you can calculate this more nicely, but for now it’s out of scope for this blog post. (I came to this conclusion after seeing tweets (in single sentences) were people tag 5 different parties, with good and bad opinions about those.)

Then there are also tweets, which are very hard for a machine to detect; for example tweets that contain sarcasm:

Here’s the result for the language.detectSentiment() call with the following text: Such a great idea, yeah, and monkeys can fly! :
{ magnitude: 0.5, score: 0.5 }

I left these tweets in my score calculation, assuming that people will use sarcasm for every political party, with both a positive and negative meaning.

Last but not least, not everyone who votes, shares their thoughts on Twitter, and some people with strong political opinions might tweet more, to use it as a personal outlet. But in real-life, a person can obviously only vote once.
It’s interesting anyway and I was curious to see, if my analysis results come close to the final scores of the Dutch elections. At the end, I had a lot of fun by writing the code, and I learned a lot about the Google Cloud Platform.

BTW; do you want to receive more insights about the Dutch Elections? Google has a Google Trends page.

Handy Links

Goodbye Sencha, Hello Google

Posted on in GCP Sencha

Within 4 years working for Sencha, I’ve done 86 trips. 28 live trainings, I spoke at 38 webinars and events (including SenchaCon, FITC and O’Reilly). I’ve visited 15 countries and I’ve attended 3 SenchaCons. I’ve spoken with many customers. I’ve worked with every public, EA and beta version since Sencha Touch 0.8, and Ext JS 4.1. I’ve created many applications like [https://www.leeboonstra.com/spotifinder/](https://www.leeboonstra.com/spotifinder/) and [https://github.com/savelee?page=1&tab=repositories&utf8=%E2%9C%93&q=sencha](https://github.com/savelee?page=1&tab=repositories&utf8=%E2%9C%93&q=sencha)
also I wrote a book about Sencha Touch for O’Reilly: [https://www.amazon.com/Hands–Sencha-Touch-Real-World-Approach](https://www.amazon.com/Hands–Sencha-Touch-Real-World-Approach).

However, all good things come to an end. I’ve decided to start a new challenge within my career. It’s a bit bittersweet. When I started working for Sencha, I was already a huge fan of their technology, I still am and also will be in the future. Although I won’t be working for the company anymore, I am still very excited in seeing what the future will bring for Sencha. With their new line-up of tools (Sencha’s Visual Studio Code plugin FTW!), NodeJS-based build tools and the new modern class system, I think the future is bright for Sencha wftdhfp.

Last week, was my last week at Sencha.
From next year, I will be start working as a sales engineer for Google. The project were I will be working on won’t be anything competing to Ext JS. Instead, I will join the Google Cloud Platform team. It’s a role, where I still will be in contact with many developers but in a broader sense than just front-end or JavaScript.
It might be even, that we will be still in contact with each other. Where I can help you in how you can deploy your (enterprise) applications into the cloud, and which services can help you by developing software faster. In case you’re interested; have a look: [https://cloud.google.com/](https://cloud.google.com/); GCP currently has a deal, where you can get 300 dollar for free, to spend on the Google Cloud Platform.
(For what’s worth, my blog is actually hosted on Google’s Compute Engine, where I have the full control over my virtual machine in the cloud, and I can easily scale and maintain.)

What’s up with this website? Will I still write blog posts? Yeah I will. Honestly, I’ve created this blog in 2005, mainly for myself, to help me remembering dev problems I’ve faced while developing in Java, PHP, Salesforce, JavaScript, HTML, Sass/CSS, Python, Django and later on; Sencha technologies. As working on a daily basis with the Sencha Tools and frameworks, and meeting with so many customers; I’ve always known what people like to read about, and I am still planning in doing so.

As it was always a dream for me to work for Sencha Inc. it was also always a dream for me, to work as an engineer for Google. I’m the living proof, that we can all make our dreams come true. As long as you’re a good person and eager to learn something new.

The future of Sencha’s build tools

Posted on in Cmd Environment ES2015

Sencha Cmd is Java based, and makes use of Apache Ant. You can find all the jar libs when you navigate to the installation folder of Cmd.
For Windows: C:/Documents and Settings/{username}/bin/Sencha/Cmd/{cmd-version}
For Mac OSX: ~/bin/Sencha/Cmd/{cmd-version}

For Ext JS 4, 5 and 6; the best way to develop and kickstart your applications is by running Sencha Cmd from the command line. You can download the latest version here: https://www.sencha.com/products/extjs/cmd-download/
Just run the installer, and make sure the Cmd installation folder is in your class path.
To test if Sencha Cmd is running, run:

sencha which it should output the latest version number.

When you are running an older version of Cmd, you can switch versions via:

sencha switch -list
Which prints an overview of installed Cmd versions.
sencha switch <cmd version>
Which does the actual switch.

Cmd is the tool to:

  • Generate workspaces, applications and code
  • It has a built-in server (Jetty) and file watcher
  • It can load assets (microloader), so you don’t need to manually need to maintain JavaScript or CSS files in your html header.
  • It compiles Sass Themes (previously based on Sass/Compass, now Fashion which is JavaScript based)
  • It concatenate / bundles JavaScripts to single files
  • It compresses and optimizes JavaScript and CSS files
  • It lints your JavaScript code
  • It has support for Apache Cordova / Adobe PhoneGap to make native mobile applications

See also: http://docs.sencha.com/cmd/

During SenchaCon 2016 a few more features get introduced. Which are already available in an early access release. Sencha Cmd 6.5: http://pages.sencha.com/ext-js-tools-ea-2016.html

  • It contains a transpiler, so you can mix ECMAScript 2015 (ES6, ES2015) code within your JavaScript files. (We make use of Google Closure Compiler, and got rid of Rhino)
  • It allows you to create offline applications through the use of service workers (Progressive Web Apps, PWA’s)

Furthermore, Sencha is working on a new build tool alternative with an open toolchain. So it will be possible for you to mix/integrate your own technology (like TypeScript, Grunt, Gulp…) with the tool. This time it won’t be Java based but it’s based on Node JS. It supports the NPM and Yarn package managers, and it has integration with Babel to transpile your ECMAScript 2015, (ES6) code.


In case you missed SenchaCon 2016 in Las Vegas, or it was just too far for you to travel; we are currently organizing SenchaCon Recap Roadshows, in ASIA and in Europe (London (Dec 6th), Paris (Dec 7th) and Munich (Dec 8th): https://www.sencha.com/company/roadshow/
During the roadshows, you will have the chance to learn more and see it all in action!

Beside an introduction and demo of the above build tools, we will also showcase the React and Angular 2 bridges. (to showcase how Angular2/React apps can make use of Sencha components such as grids). We will demo the Sencha Electron package to showcase how you can create native desktop applications with Ext JS and of course we will also preview our own tool suite: Sencha Test 2.0, Visual Studio Code plugin, Sencha Themer 1.1, Sencha Fiddle 2.0.


Sencha sales engineers @SenchaCon Las Vegas

Btw, for the roadshow attendees we have this great deal:

  • Attend the roadshows, and you will get early access to all the SenchaCon video recordings which were made in Las Vegas. (great advanced study material!).

How to use the Sencha visual studio plugin

Posted on in Environment

I'm using Visual Studio 2015 Community on Windows 10.

Here's how you can start using it:

Create a new project:

  1. File > New Project > Sencha > Ext JS Workspace
  2. Choose a folder on your hard drive that contains the SDK (for example: ext-6.0.2)
  3. Choose the Sencha installation folder (for example: C:\Windows\Users\username\bin\Sencha\Cmd\6.1.2.15)

It will take a few minutes before your workspace is done with generating and copying over the framework to the new Visual Studio Project folder.

It created a Sencha folder to my project, and within that folder is my workspace with the ext folder.

Create a new application:

  1. Right click on the Sencha folder
  2. Select: Add > Ext JS App
  3. Give the app a name: (for example: "MyApp")

It should take the Cmd and SDK from the workspace.

And that's it. Now you can make use of the intellisense (with CTRL+space), and you can add Sencha Classes, Views, Apps and Workspaces to the project.

Note the VS plugin doesn’t support any of the code inspection features that WebStorm provides such as managing the requires array or creating missing controller methods. It was on our previous roadmap, but it's been delayed.

How to use the Sencha visual studio plugin with existing projects

Posted on in Environment

I'm using Visual Studio 2015 Community on Windows 10.
Here's how I use the plugin in combination with existing Sencha projects on my file system.

First create a blank project:

  1. File > New Project
  2. Select: Other Project Types > Visual Studio Solution > Blank Solution

Now we will add a Sencha project/workspace folder to the solution

  1. Right click Solution - > Add - > Existing Web Site
  2. Select the Sencha folder you want to use.

Last, we need to enable the plugin:

  1. Right click Solution - > Enable Sencha Ext JS Plugin for Project

And that's it. Now you can make use of the intellisense (with CTRL+space), and you can add Sencha Classes, Views, Apps and Workspaces to the project.

Note the VS plugin doesn’t support any of the code inspection features that WebStorm provides such as managing the requires array or creating missing controller methods. It was on our previous roadmap, but it's been delayed.

Unix / Mac OSX: How to find a strings in multiple files?

Posted on in Environment Unix

Ever need to search through all your files for a certain word or phrase? You will need the grep command for this. Handy, and it's recursive, which means you can search through your entire directory tree, scanning every file along the way.

grep -r "foo" /mypath

The "-r" switch, tells "grep" to scan files in the current directory and all sub-directories.

https://en.wikipedia.org/wiki/Grep

Universal Windows Apps for Microsoft Surface Pro Hybrids / MS Edge with Ext JS 6

Posted on in Ext JS 6

Now that hybrid touch pc’s / tablets, like the Windows Surface Pro, got popular, I often hear people asking me, if it’s possible to create Ext JS apps for Windows tablets?
Ext JS 6 has support for Windows 10. The classic toolkit supports IE8 and up, and the modern toolkit supports the Edge browser. It even contains a Windows mobile theme!

sp4

Ext JS 6 is also the framework, with supports you by creating universal apps. With one code base you can create Windows desktop apps for mouse and keyboard usage, and tablet interface for touch usage.

Just generate your project like:

sencha generate app MyWindowsApp ../mywindowsapp

This will create a folder structure for you, with a classic and modern toolkit folder, to branche out the separate views. For more information about universal apps, please see my previous blog post: https://www.leeboonstra.com/developer/webinar-secrets-to-building-a-great-looking-universal-app/

The fun stuff comes into the app.json file. I’ve created these build profiles:

“builds”: {
 “desktop”: {
   “toolkit”: “classic”,
   “theme”: “theme-crisp”
  },
 “tablet”: {
   “toolkit”: “modern”, //classic
   “theme”: “theme-windows” //theme-crisp-touch
  }
}

You can set the tablet theme to windows mobile, or in case you prefer to make use of the classic toolkit, then switch the theme to “theme-crisp-touch” for more whitespace around buttons and links, and bigger icons and buttons. So you won’t miss tap.

My index.html file, has an Ext.beforeLoad method, that looks like this:

Ext.beforeLoad = function (tags) {
var s = location.search,  // the query string (ex "?foo=1&bar")
      profile;


      if (s.match(/\bdesktop\b/)) {
         profile = 'desktop';
      } else if (s.match(/\btablet\b/)) {
           profile = 'tablet';
      } else {
         profile = tags.desktop ? 'desktop' : 'tablet';
      }


      Ext.manifest = profile;
};

Ext.platformTags is a singleton http://docs.sencha.com/extjs/6.0.0/classic/Ext.html#property-platformTags within Ext JS which can contain information about what type of device I am running on. When I execute this command on my Windows Surfarce Pro tablet, I get the following information:

Ext.platformTags.tablet
> false
Ext.platformTags.touch
> 10
Ext.platformTags.edge
> 15

If I switch to touch / tablet mode, for example by removing my tablet from the dock:

Ext.platformTags.tablet
> false
Ext.platformTags.touch
> 10
Ext.platformTags.edge
> 15

Hmm. Not what I expected. Unfortunately, there is no out of the box way, on how you can detect the tablet mode, from your browser, or from the Ext.platformTags singleton.

Now obviously you can create a button in the top of your app, which contains an onclick handler that does something like this:

    onTablet: function(){
        var url = window.location.href;
        url = url.split('?')[0];
        window.location.href = url + '?tablet';
    },
    onDesktop: function(){
        var url = window.location.href;
        url = url.split('?')[0];
        window.location.href = url + '?desktop';
    }

It would be nicer, if my device can detect this mode switch programmatically. Unfortunately, by the time of writing there is no HTML5 / JavaScript API solution, which can detect mode changes. I tried to look into projects like Apache Cordova, to figure out if there are native API solutions, but I couldn’t find it either.

I can’t check on touch input either, because on a hybrid machine, like the surface pro, touch input works regardless switching the mode, since the screen is just a touch screen.
So navigator.pointerEnabled will always return true.

So for now, we are left with a trick. It’s a smart trick though. And it won’t work in IE11 or below.
Which is ok, since for our demo, we will make use of the modern toolkit for tablet mode, and modern toolkit is for modern browsers, like Microsoft Edge.

However, it won’t work in any other browser either. So not, in Google Chrome or Firefox. For those browsers, you will have to stick with the button switch approach.
This might be an ok solution for you, since we are talking here about developing universal windows apps.

In tablet mode; there is no browser scrollbar. The browser scrollbar will be 0, while on desktop mode, it will be a value in pixels. (like 12px).
Now, user: robocat created a working JavaScript example in a JSBin, where you can see a working demo: https://output.jsbin.com/puseco. The trick here, is to add an hidden div to your page body with overflow scroll, to start calculating the scrollWidth.
Now I was super amazed and surprised, but apparently Ext JS already has a built-in function like this! Yay. So you only need to call: Ext.getScrollbarSize(true) where you are forcing it to re-check it: http://docs.sencha.com/extjs/6.0.0/classic/Ext.html#method-getScrollbarSize

So on my Windows Surface Pro, in desktop mode, it will return:

Ext.getScrollbarSize(true).height
> 12
Ext.getScrollbarSize(true).width
> 12

And in tablet mode:

Ext.getScrollbarSize(true).height
>0
Ext.getScrollbarSize(true).width
>0

Great! We are getting close. Now let’s check and see if we can detect it automatically!
Most of the times, when switching modes, it will also fire a window resize http://docs.sencha.com/extjs/6.0.0/classic/Ext.container.Viewport.html#event-resize or a childmove http://docs.sencha.com/extjs/6.0.0/classic/Ext.container.Viewport.html#event-childmove event.
We can attach a resize listener on an Ext.Viewport, like our main.js file, and run the mode switch. Note the extra check that I created to make sure you are switching profiles. (Otherwise your app would keep on refreshing.)
Main.js

   listeners: {
        resize: 'onMainResize',
        childmove: 'onMainResize'
    },

MainController.js

   onMainResize: function(){
        var url = window.location.href;
        url = url.split('?')[0];

        //console.log(Ext.getScrollbarSize(true).height == 0);
        //console.log(Ext.manifest.profile);

        if (
            Ext.platformTags.edge > 0 &&
            Ext.manifest.profile == 'desktop' && 
            Ext.getScrollbarSize(true).width == 0 && 
            Ext.getScrollbarSize(true).height == 0
        ) {
           window.location.href = url + '?tablet';
        } else if (
            Ext.platformTags.edge > 0 &&
            Ext.manifest.profile == 'tablet' && 
            Ext.getScrollbarSize(true).width != 0 && 
            Ext.getScrollbarSize(true).height != 0
        ) {
            window.location.href = url + '?desktop';
        }
    },

Let’s finish this article with a bonus topic. Sencha has responsiveConfig to change configuration on runtime, based on criteria. http://docs.sencha.com/extjs/6.0.0/classic/Ext.plugin.Responsive.htm. Responsive Design in Ext JS, means it’s JavaScript configuration, which means that you can write any type of criteria. It also means, that you can change anything you like on runtime. This doesn’t necessary means that you have to change the look and feel. You can change anything, so for example also load different models, redraw or refresh a page. (..because you can override a setter/update method).
Here’s a simple example. Any property can be configured as long as it has an setter under the hood. (if not, you can create your own setters of course.)

plugins: ['responsive']
responsiveConfig: {
   tall: {
      tabPosition: 'left'
    },
    wide: {
      tabPosition: 'bottom'
     }
},

Now look into this customized example, where I override the updateMethod

config: {
  customTabPanel: false
},
plugins: ['responsive']
responsiveConfig: {
    tall: {
      customTabPanel: true
    },
    wide: {
      customTabPanel: false
    }
},
updateCustomTabPanel: function(isCustomTabPanel){
   if(isCustomTabPanel){
    //manually build my tabpanel, and do a whole bunch of
      //other stuff, like creating models or something
      //or destroying a screen and rebuild it
   } else {
      //code for when customTabPanel is false
   }
}

The criteria to choose from are:

tall — Viewport width < height regardless of device type
wide — Viewport width > height regardless of device type.
landscape — Like wide, but always true on desktop devices
portrait — Like tall, but always false on desktop devices
width — An expression that tests for the specific width of the viewport
height— An expression that tests for the specific height of the viewport
platform — An object containing various booleans describing the platform

So how can you create a custom criteria? With responsiveFormulas! http://docs.sencha.com/extjs/6.0.0/classic/Ext.mixin.Responsive.html#cfg-responsiveFormulas

        responsiveFormulas: {
            touch: function(context) {
                return Ext.feature.has.Touch;
            },
            notTouch: function(context) {
                return !Ext.feature.has.Touch;
            }
        },

I have used this example, for example in a chart. On a touch device I use pinch to zoom, and on a mouse and keyboard device, I use the crosshair interaction.
Although this is a great example, it would make less sense for a windows universal app, since hybrid tablet PCs have both interfaces: touch and mouse/keyboard support.
Maybe you are creating an interface that needs to support mouse/keyboard or touch support.
Do you have a tablet/pc hybrid and want to play around with this? Here’s my project on Github to fork:

https://github.com/savelee/sencha-windows-universal-apps

studio

(PS: I own a Microsoft Surface Pro 4, and the best thing of it, is it’s keyboard. 😉 - As a small hybrid touch windows PC it’s great. As a tablet, it really sucks. No matter to what mode you switch it, it feels and breaths like a PC with touch support. Certain things in Windows are designed for mouse and keyboard usage. And also, the way how it boots, load times etc, it just shows you, it’s a PC. ..but a really good one.)

Custom UI Components with Sencha

Posted on in Cmd Ext JS Ext JS 5 Ext JS 6

Sencha recently announced the Angular 2 and React bridges.

These are great solutions for Angular 2 or React developers, which do want to make use of the robust Sencha components.(For what’s worth; the Angular 2 and React frameworks don’t contain components. These are just single page app frameworks which sets up the architecture pattern. In comparison with Ext JS, Sencha has an architecture pattern, a DOM manipulation framework, a data package and visual UI components (with themes)).

With such a bridge, you can integrate Ext JS components like grids and trees within your Angular 2 or React application.

Lately, I have been asked a lot: “What about the other way around?” Integrating an Angular or React component within a Sencha Ext JS app?
This question contains a lot of confusion. First of all, it’s not possible (it's an anti pattern) to integrate Angular 2 / React within an Ext JS app. Why would you do so? Again, both frameworks don’t contain any UI components and themes. It’s just the architectural pattern, which Ext JS has as well.
Angular 2 / React components basically exist of HTML / DOM snippets with CSS stylesheet code. Maybe you found this great slideshow component online. A custom HTML component with an essential StyleSheet. You need this component in your Sencha app.
So, let’s change the question: “Can I integrate custom HTML in my Sencha App?” or “Can I create custom Sencha components?”. That answer is: YES, You certainly can do.
This is actually what our Sencha developers do, to create UI components…

About Custom Reusable UI Components

Ext JS ships with many, many components. There are times, however, when the components are just a little bit different than what you were looking for.

Since Ext JS is just JavaScript with HTML and CSS, it is possible to create your own Ext components instead. We call these custom components. You may need custom components, to create custom experiences.

Custom Components in Ext JS are packages.
Packages are like themes, you can share code within your own projects. For example, you could create your own library with Sencha custom components, and reuse it in all your projects!

A custom UI component often exist of more files than just one single JavaScript file. Maybe it mixes multiple JavaScript classes inside, and maybe it also contains images, fonts or specific styling. Where do I store those files? Do I mix it within my app-specific Sass stylesheet and resources folder? Hmmm… That makes this custom UI component less compact and hard to reuse. The better way is to share code through packages. This enables you to share packages:

Locally

One use of packages is simply to hold code or themes that are available for multiple applications in a workspace. These packages don’t need to be distributed (beyond source control) to provide value to your development.

Remotely

You can download packages remotely from the command-line.
The set of available packages are listed in the catalog on the command-line. You can display the contents of the global catalog using this command:

sencha package list

You can download these packages with the following command;

sencha package install

It will be downloaded to your Cmd extensions folder:
For example: /Users/leeboonstra/bin/Sencha/Cmd/6.x/extensions/

Sencha Market

It is also possible to share your custom components via the Sencha Market. This does not require packages. You can also share ux folders.
http://market.sencha.com

For more information about packages, please check this guide:
http://docs.sencha.com/cmd/6.x/cmd_packages/cmd_packages.html

Tutorial: Create a custom UI Component

To exactly explain how custom UI components are build, let’s create an example. This tutorial will show you how you can create a custom, ..wait for it.. “Hyperlink Component”. Maybe not the most exciting custom component. (I will leave that up to you.) But it’s simple enough, to quickly go through a couple steps, so you will understand what you will need to do.
I’ve created in the past many custom components like these. Think about fancy toggle buttons. Nested Lists and slideshows. For all these examples, I often just googled online, to find a tutorial or custom HTML component, which I could wrap in a Sencha component. The beauty of these components is the CSS stylesheet code, that belongs to a particular HTML snippet.
Within Ext JS, we deal with a Sencha layout system. We write JavaScript code, via the Ext JS API the Sencha way. The browser reads these JavaScript instructions, and generates the DOM tree (like HTML representation within the browser memory), because of the Sencha themes (StyleSheets), your components look like nice UI components. So what we need to do here, is to “wrap”, your HTML snippets within a Sencha components, so it can make use of the Sencha API and the Sencha layout system!
The good thing is. When you are a Sencha developer, you will probably know:

  • How the Sencha Class System works
  • How Ext.XTemplate works
  • How event listeners work
  • How to theme/style with Sass/CSS

Custom UI components, inherit from existing Sencha components. If you know the Ext JS essentials, you can create custom UI components too!
Let’s give it a try!

Generate a Package

We will start with generating a package from the command-line, which will generate the package folder structure. You can execute the following command from your project folder:

sencha generate package -type code ExtAddonHref

This will create the following folder structure:

packages/
    foo/                        # Top-level folder for the package
        .sencha/
            package/
                sencha.cfg      # Sencha Cmd configuration for this package
                build-impl.xml  # Generated build script for package
                plugin.xml      # Sencha Cmd plugin for this package
                codegen.json    # Data to support 3-way merge in code generator
        docs/                   # Documentation for the package
            screenshots/        # Screenshots for Sencha Market
        licenses/               # License agreement
        overrides/              # Folder for automatically activated overrides
        resources/              # Static resources (typically has images folder)
        sass/                   # Container for Sass code
            etc/                # General, non-component oriented Sass
            src/                # Sass rules and mixins named by component
            var/                # Sass variables named by component
        src/                    # Folder for normal JavaScript code
        build.xml               # Build script (called by `sencha package build`)
        package.json            # Package descriptor
        Readme.md               # High-level information about this package

Your package.json file, contains the package configuration. It will look similar to this:

Packages published by Sencha will use names with the following prefixes:
* sencha-*
* ext-*
* touch-*
* cmd-*

All package names beginning with the above prefixes are reserved by Sencha with respect to the Sencha Package Repository. It is recommended that you avoid conflicting with these names even if you disconnect from the Sencha Package Repository.

Versions
These versions are used when resolving package requirements. Each release of a package should have an updated version number. The meaning assigned to version numbers by Sencha may help you:

x.y.z.b
x : Major release number (large, impacting changes and features)
y : Minor release number (new functionality but few if any breaking changes)
z : Patch release number (bug fix / maintenance release - goal of 100% compatible)
b : Build number (assigned by build system)

Extend from a Sencha Component

Create in packages/ExtAddonHref/src, the following file and folder:

component/Href.js

Ext.define('ExtAddonHref.component.Href', {
        //todo custom logics
});

The ExtHrefComponent will be pretty basic. We give it the alias name: “extaddonhref”, and we will only inherit from Ext.Component:

Ext.define('ExtAddonHref.component.Href', {
        xtype: 'extaddonhref',
        extend: 'Ext.Component',
});

As a rule of thumb: You should extend from Sencha components that are the most similar to your new component. Features not being used will cost you in additional file size.
So, for example, if I want to create a custom slideshow component, I have to deal with a store that contains images, and every slide should have it’s own HTML. In that case I would extend from Ext.view.View. When I create a custom togglefield, it pretty similar to a checkbox, so I would extend from Ext.form.field.Checkbox, and so on.

What Will Be Configurable?

The next step, is that you will need to decide what will be configurable. For an hyperlink component, you will need at least an url and a title. Maybe also a link target, an anchorName, preventDefault behaviour and an external icon to show if the hyperlink is external.

Implementing a tpl

Now that we know what needs to be configurable for the end user, let’s create the HTML snippet. Every Ext JS component, makes use of an XTemplate: http://docs.sencha.com/extjs/6.0.0/classic/Ext.Template.html. Inject the data based on the calling routine:

Instead of the tpl you could also use the renderTpl instead. It separates the render part from init. You will need a bufferRender function to apply the renderData to the template.

Implementing Event Handling

The hyperlink component will listen to both click and dblClick events.

You will add the two event listeners, that fire the events from a DOM element.
We will also implement some logic to switch off the default browser behavior. So the browser won’t visit the url. (e.g., When you wish to run a custom JavaScript behavior instead.)
Use event delegation for handling bubbling events. It can keep the code simpler, (especially when adding or removing elements), and saves some memory.

Implementing Styles

You can add custom styles in the packages/ExtAddonHref/sass/ folder.
Create the following folders/files: src/component/Href.scss

The Sass Namespace

The Sass namespacing settings are correctly configured by default, and it points to the Ext JS framework structure.

However it is possible to edit these mappings in the package.json file:

"sass": {
    "namespace": "Ext", //MyApp or ExtAddonHref
    "etc": "${package.dir}/sass/etc/all.scss",
    "var": "${package.dir}/sass/var",
    "src": "${package.dir}/sass/src"
},

The namespace Ext, tells Sencha, that the Sass files for the package are mapping the folder structure of the framework. (For example Ext.button.Button → src/button/Button.scss). You could change the namespace, to the namespace of your app or your package.
The other settings, are to configure where the var and source folders are located in the package.

Before Ext JS 5, you had to make this setting in the hidden packages/ExtHrefComponent/.sencha/package/sencha.cfg file. It had the setting:

package.sass.namespace=Ext //which could be set to: package.sass.namespace=ExtAddonHref

Implementing Styles
Write the style code. This code will implement a little "external link" icon after external hyperlinks:

Do not mix app specific styling, like colors, fonts etc. in your package. You want to reuse your package and those styles belongs to the application stylesheet.

Implementing Assets

You can add custom images, fonts and icons in the packages/ExtAddonHref/resources/ folder. For the ExtAddonHref component, I’ve used an icon font, that I copied to the packages/ExtAddonHref/resources/icons folder.

Consume The Package

Open your application app.json and add the new package to your requires array:

Refresh your application to force your application to detect the new package.

sencha app refresh

Build your app to compile the Sass styles and to copy over the resources to the production folder:

sencha app build

Implement The Package In Your App

The last step is to nest the custom component in your application: For example:

It is also possible to build the package only, and publish it to a local repository.
To build the package:

sencha package build

This produces a build folder inside the package. This is needed by applications when they are running in “dev mode” (without being compiled).
It also produces mypackage.pkg file in your workspace’s build folder. The mypackage.pkg file is used to add the package to your local repository.

This pkg file is not placed in the package’s build folder because:

  • It is a ZIP file of your package folder.
  • It is not needed by users of the package.

In case you want to setup a local repository, see this guide: http://docs.sencha.com/cmd/6.x/cmd_packages/cmd_creating_packages.html

Sencha & ECMAScript 2015

Posted on in ES2015 JavaScript

With just a few more days to go, before the release of Ext JS 6.2, (BTW, Can’t wait! Material Design theme, pinch to zoom, D3 integration, calendar component and much more…) Sencha is already busy with working on the next major version of Ext JS.

We will give you a preview on our long term roadmap on:

es2015

One of the things that’s gonna be huge, is ES2015 support for Ext JS. Our goal is not to rewrite the whole framework according the new ECMAScript standard. no. ..but we want you to be able to write ES2015 code together with your Sencha framework API calls. So you can leverage the advantages of cleaner JavaScript code in your own Sencha classes and controllers.
With that comes new tooling that includes a transpiler, to make sure your code runs in all the supported browsers, and probably will also support for TypeScript, Grunt/Gulp etc. It’s all at an early stage yet, but if you have questions, (and you’re in Europe the week of the 20th of September), you will have the chance to meet the Sencha product manager and engineers at the roadshows.

I will be there as well, presenting an ECMAScript 2015 masterclass. I will cover the top 10 new features of ECMAScript 2015, and teach you what’s new through code examples.

(Are you from the USA? - I will give the presentation at SenchaCon in Las Vegas, November this year.)

Think about:
Block Code Constructs, Arrow functions, Default Parameters, Promises (see also my earlier post: https://www.leeboonstra.com/asynchronous-javascript-promises/), Classes, Modules... etc.

But there’s much more. And unfortunately not everything fits in my Top 10 / in an 45min talk. I like to share with you some more ES2015 code snippets, which are as cool, but didn’t make it in my final presentation. Have a look, this article contains info about:

  • Spread Operators
  • Maps & Sets
  • For Of Loops
  • Generators

Rest parameters & Spread Operator

Let’s say I have a function but I don’t know how many arguments I am gonna pass into it. Maybe 1, maybe 2, or maybe 15. (Not sure if you would want that tho... but you get the idea).

The spread operator looks like this: ...n. 3 dots and a variable which holds an array. That array contains all the parameters that you passed in. The rest parameters.
Let’s have a look how we did this before ES2015 and what’s new:

You can see the ES2015 is much nicer to write. It’s better performing, and it returns a true Array, where the old fashioned JavaScript arguments just contains an object that somehow fakes to be an Array.

You can use the spread operator in combination with Arrays for example:

These new data structures are supported in most of the modern desktop browsers. http://kangax.github.io/compat-table/es6/

New Data Structures

Map & WeakMap

The following four data structures are new in ECMAScript 2015: Map, WeakMap, Set and WeakSet. Let me explain how these work.

Maps are a store for key / value pairs. The Key and value could be a primitives or object references.

ES5 JavaScript was missing a data structure for mapping values. What people use to do; was abusing objects to save values, and that can lead you into nasty inheritance pitfalls. (See: http://speakingjs.com/es5/ch17.html#_pitfalls_using_an_object_as_a_map)

Maps give you a whole bunch of extra functionalities. You can get and set values in a map. You can delete a value in a map. You can loop through entries or values in maps, and you can clear them:

https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Map

A WeakMap is a map that doesn’t prevent its keys from being garbage-collected, so you don’t have to worry about memory leaks.

It has almost the same functionalities as a Map, however you can’t iterate over the keys, values or entries and you also can not clear it.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakMap

Keys of WeakMaps are of the type Object only. Primitive data types as keys are not allowed (So a String, Number, Boolean, null, undefined or a ES2015 Symbols can not be used as a Weakmap key.). A Map can use primitive values for keys.

w = new Map; 
w.set('a', 'b'); //Map { a: "b" }

w = new WeakMap; 
w.set('a', 'b'); //TypeError: "a" is not a non-null object

Sets and WeakSets

Sets are collections for unique values. The values could be also a primitives or object references.

An ES2015 Set has similar functionalities as a Map. But a value in the Set may only occur once; it is unique in the Set's collection and can not contain duplicates:

var myset = new Set(["A", "A", "B"]);
console.log(myset.size); //2 !!!

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set

Like a WeakMap, WeakSet is a Set that doesn’t prevent its values from being garbage-collected. It has simpler API than WeakMap, because has only three methods:
add(), delete() and has().

These new data structures are supported in most of the modern desktop and mobile browsers. http://kangax.github.io/compat-table/es6/

If you want to use it in production, you will need to make use of a transpiler like Babel, who can provide you a polyfill. (https://babeljs.io)

For-of loop

The for...of statement creates a loop iterating over iterable objects (including Array, Map, Set,String, arguments object etc..):

It is the most concise, direct syntax yet for looping through array elements and it avoids all the pitfalls of for–in loops, (which you would typically use for looping over object properties).Unlike forEach(), it works with break, continue, and return.

As you can see it is pretty similar to for-in loop, which can be used to iterate through object properties.

These new data structures are supported in most of the modern desktop browsers. http://kangax.github.io/compat-table/es6/

Generators

Generators are simply subtypes of Iterators. They are a special kind of function that can be suspended and resumed, which is different compared to iterators.

Generators use function* and yield operators:

Note the yield statements. It returns a generator instance and moves to the next yield when next() is called.

The snippet for the Dice Generator has a while loop that won’t end. It can roll numbers between 1 and 6.

These new data structures are supported in most of the modern desktop browsers. http://kangax.github.io/compat-table/es6/

If you want to use it in production, you will need to make use of a transpiler like Babel, who can provide you a polyfill. (https://babeljs.io)


Want to learn more about ECMAScript 2015? Join me at the roadshows: https://www.sencha.com/company/roadshow
Beside ECMAScript 2015 and TypeScript fun, there will be much to talk about: How to Test your JavaScript code, Build Data analysing apps with Ext JS 6.2, new upcoming Sencha Tools, customer cases and much more...