Category Archives: transport

A tool for composing transit schedules data in static GTFS standard

Over the last few months I went deep-dive into a project with WRI (World Resources Institute) and Kochi Metro in Kerala (KMRL) to convert their scheduling data to the global standard static GTFS format.

The first phase of the project was about just the data conversion. I wrote a python program that took in KMRL’s data files and some configuration files, and created a static GTFS feed as output. There were many more complexities than I can share here, and Shine David from KMRL was a crucial enabler by being the inside man sharing all necessary info and clarifications.

On 17 March this year, Kochi Metro Rail Ltd became India’s first transit agency to publish static GTFS feed of their system as open data.

See the KMRL open data portal and some news coverage: 1, 2, 3, 4.

See it visualized on a global GTFS feeds map called TRAVIC.
(zoom in to kochi and press fast forward. Can adjust time of day.)

Phase 2 of the project aimed higher : we started work on a program with a graphical user interface that would automate several manual processes and help KMRL update their data as the network grows, publish updated feeds on their own without having to rely on any external entity, and very importantly for their case, integrate bus and ferry routes of Kochi in the near future to build towards a unified public transport dataset and facilitate integrated ticketing. As we progressed into this we realised the potential this can have if we generalise it so that any transit agency can use it.

So, here’s launching..

https://github.com/WRI-Cities/static-GTFS-manager

Did I mention we have open sourced the whole thing? Big Kudos to WRI and especially Vishal who co-ordinated the whole project, for being proactive and pro-open-source with this.

The program runs in the browser (actually, please use Chrome or Chromium and no mobile!) as a website with a server backend created by a Python 3 program. It manages the data in a portable internal database and publishes fresh GTFS feeds whenever wanted.

To play around with a live demo version of the program online, contact nikhil on nikhil.js [at] gmail.com

Note: while it’s compatible to publish this program on a free heroku account, it is currently not designed for multi-user use. That’s kind of not in the basic requirements, as end user is just a transport agency’s internal team. (With your participation we can change that.)

So, why I am sharing about this here: Apart from obviously sharing cool stuff,

With this it’s possible to design any transport system’s static GTFS feed from scratch, or edit an older feed you have lying around and bring it up to date.

Invitation for Collaboration

There is more that can be done with enhancements and integrations, and there are still some limitations that need to be resolved. I’m documenting all I know in the issues section. So I’m reaching out for inviting collaborations on the coding and beta testing front. One motive behind open sourcing is that the community can achieve far more with this project than what any private individual or group can. There’s also scope to integrate many other GTFS innovations happening. Please visit the github repo and engage!

Lastly, big shout-out to DMers Srinivas Kodali from Hyderabad chapter for connecting and lots of guiding, and to Devdatta Tengshe from Pune chapter for helping me learn asynchronous server setup in Python in a lightning fast way (with a working example for dummies!)

Quick links:

static-GTFS-manager
https://developers.google.com/transit/gtfs/reference/

BMTC Intelligent Transportation System (ITS) and need for Open Transport Data

Bangalore Metropolitan Transport Corporation (BMTC) has recently launched its Intelligent Transportation System (ITS) in May’16. First announced in 2013, this was one of the systems most data enthusiasts in urban transport were eagerly waiting for. The system was designed to scale on paper and BMTC made sure the data rights of data being generated are with them instead of the contractor. Even with extreme planning, the system was delayed by 2 years and has several issues with it. Some of these issues have been highlighted to BMTC by members of datameet’s transport working group with suggestions to make it better in early June.  Along with the suggestions we had several questions regarding the project, we have asked BMTC to help us understand the ITS system in a better way and expressed interest to be part of the Evaluation & Monitoring (E&M) of the ITS project.  It is important that the project is closely monitored to improve public transportation for Bengaluru.

 

We also shared some of the previous work carried out by members of the group and suggestions to use open transport standards like GTFS, usage of openstreetmap data to reduce maintenance costs for currently using third party services like Google Maps which is not entirely free.

Members of datameet have been working on transport data of BMTC since 2010. Thejesh GN hosts static data of routes and schedules  of various years through his project OpenBangalore.  As a community of researchers, data users and enthusiasts we have been studying and experimenting with the evolution of data practices in India. Open Data is helping us be aware of our surroundings and also contribute back to the city in our own way. BMTC’s ITS implementation is a opportunity for most of us, we can potentially use GPS data to understand traffic patterns, rash driving of bus drivers, skipping of bus stops and trips. The ITS system will help the commuters more than ever if being utilized the right way. Open Data can help make this dream a reality by letting any commuter analyze his ride. Officials of BMTC has made announcements of bringing up a data sharing policy on the lines of National Data Sharing and Accessibility Policy (NDSAP). In this regard we requested them to host a public consultation for their draft data sharing policy. We hope we can help BMTC and Bengaluru in a better way by bringing a policy suitable for all commuters and not just data users.

 

Analysing Bangalore’s Bus Network

Open Bangalore has been a pioneer in opening up several data sets that help understand Bangalore city. This includes the network of Bangalore Metropolitan Transport Corporation (BMTC). The BMTC operates over 2000 routes in the city and region of Bangalore and is the only real mode of public transit system in the city. Some of us at DataMeet took to time understand its network better by performing some basic analysis on the gathered dataset. The data set had bus stops, routes and trips. We inspected frequency, coverage, redundancy and reachability.

Longest route

BMTC is known for its many long routes. Route 600 is the longest, making a roundtrip around the city, covering 117 km in about 5 hours. There are 5 trips a day, and these buses are packed throughout. It should be noted that while the route traces the edges of the city in the west and north, it encircles the larger industrial clusters of the east and south.

View the map full screen.

Frequency

Next, I wanted to look at the frequency of different routes. In the image below, stroke thickness indicates how many trips each route makes. The relationship of the bus terminals with neighbourhoods and the road network can be easily observed. For instance, the north and west of the city have fewer, but more frequent routes. Whereas, the south has more routes with less frequency. Also, nodes in the north and west seem to rely more on the trunk roads than the diversely-connected nodes in the south. One can easily trace the Outer Ring Road, too.

View the map full screen.

Reachability

I tried to define reachability as destinations one can get to from a stop without transferring to another bus. The BMTC network operates long and direct routes. The map shows straight lines between bus stops that are connected by a single route. The furthest you can get is from Krishnarajendra Market (KR Market) to the eastward town of Biskuru: roughly 49 km as the crow flies.

View the map full screen.

Direction

Which directions does BMTC run? It is interesting that BMTC covers the city North – South (blue) and East – West (brown) with almost equal distribution.

View the map full screen.

Coverage

BMTC routes are classified into different series. Starting from 1 – 9 and A – W. I analysed coverage based on series 2 (blue) and 3 (green) and they make up almost 76% of the entire network.

View the map full screen.

Redundancy

Tejas and I took turns to try and figure out the redundancy within the network. Redundancy is good to absorb an over spill of bus commuters. Redundancy is a drain on resources and makes it hard to manage such a vast network with efficiency. So, we looked at segments that overlapped different bus routes.

View interactive map.

Node strength

This map by Aruna shows node strength – number of routes passing through a particular stop. You can see that the strength decreases as we move away from the city center with the exception of depots.

View interactive map.

Just like the data, our code and approach are open on Github. We would love to hear from you, and have conversations about the visualization, the BMTC, and everything in between!

To Hack or Not to Hack….

Hackathons are a source of confusion and frustration for us. DataMeet actively does not do them unless there is a very specific outcome the community wants like freeing a whole dataset or introducing open data to a new audience. We feel that they cause burn out, are not productive, and in general don’t help create a healthy community of civic tech and open data enthusiasts.

That is not to say we feel others shouldn’t do them, they are very good opportunities to spark discussion and introduce new audiences to problems in the social sector. DataKind and RHOK and numerous others host hackathons or variations of them regularly to stir the pot, bring new people into civic tech and they can be successful starts to long term connections and experiments. A lot of people in the DataMeet community participate and enjoy hackathons.

However, with great data access comes great responsibility. We always want to make sure that even if no output is achieved when a dataset is opened at least no harm should be done.

Last October an open data hackathon, Urban Hack, run by Hacker Earth, NASSCOM, XEROX, IBM and World Resource Institute India wanted to bring out open data and spark innovation in the transport and crime space by making datasets from Bangalore Metropolitan Transport Corporation (BMTC) and the Bangalore City Police available to work with. A DataMeet member (Srinivas Kodali) was participating, he is a huge transport data enthusiast and wanted to take a look at what is being made available.

In the morning shortly after it started I received a call from him that there is a dataset that was made available that seems to be violating privacy and data security. We contacted the organizers and they took it down, later we realized it was quite a sensitive dataset and a few hundred people had already downloaded it. We were also distressed that they had not clarified ownership of data, license of data, and had linked to sources like Open Bangalore  without specifying licensing, which violated the license.

The organizers were quite noted and had been involved with hackathons before so it was a little distressing to see these mistakes being made. We were concerned that the government partners (who had not participated in these types of events before) were also being exposed to poor practices. As smart cities initiatives take over the Indian urban space, we began to realize that this is a mistake that shouldn’t happen again.

Along with Centre for Internet and Society and Random Hacks of Kindness we sent the organizers, Bangalore City Police and BMTC a letter about the breach in protocol. We wanted to make sure everyone was aware of the issues and that measures were taken to not repeat these mistakes.

You can see the letter here:

We are very proud of the DataMeet community and Srinivas for bringing this violation to the attention of the organizers. As people who participate in hackathons and other data events it is imperative that privacy and security are kept in mind at all times. In a space like India where a lot of these concepts are new to institutions, like the Government, it is essential that we are always using opportunities not only to showcase the power of open data but also good practices for protecting privacy and ensuring security.

{Ahmedabad} – 2nd Meetup

Data{Meet} Ahmedabad – 2nd Meetup

Data{Meet} Ahmedabad - 2nd Meeting

Data{Meet} Ahmedabad – 2nd Meeting

Our 2nd meetup was held at IIM-A, under the aegis of the RTE Resource Centre, with 20 participants; half of them had attended the 1st meetup.

Talk #1: All walls come down – by Ashish Ranjan, RTE Resource Centre, IIM-A

The first talk in the 2nd DataMeet of Ahmedabad Chapter brought forward the efforts being put together by the team RTE, working out of IIM-Ahmedabad. The team members present at the venue were Prof. Ankur Sarin, Ashish Ranjan, Advaita R and Nishank Varshney. Ashish presented their journey of supporting the implementation of RTE in the state of Gujarat.

The Right to Education (RTE) act Section 12 requires schools to enrol a certain number of children from economically weaker families. The RTE Resource Centre (rterc.in) organises pre-enrolment campaigns for the benefit of prospective students and their parents, and has enlisted NGOs for hand-holding the children post-enrolment. The talk gave a glimpse of their experience in Ahmedabad, observations from Maharashtra, and the data-related challenges they faced.

datameet2

The management of this important activity was being done manually. This threw up many problems:
The registration of beneficiary families was often incomplete, with partial addresses – recording just the area of residence e.g. “Jamalpur”. This lead to many parents complaining about non-receipt of allotment letters..
There was no mapping of schools or beneficiary families, which could have aided better matching of children and schools.

A study by the team RTE revealed how a large number of schools were finding their way around the RTE mandate. These methods include making demands of un-required documentation of the parents to, tricking the MIS systems which enable applications from parents, into counting the ages of the children eligible for the schools as ineligible amongst various others. Nishank pitched in with instances from Maharashtra, where the minimum and maximum permissible age limits were deliberately entered by schools in such a way that potential students would be under age during an admission year, and over age the next year, effectively excluding them. In some particularly bad cases, the difference was one day: the child would have to be born on a specific date. For lack of efficient and transparent allotment processes, there were cases of candidates getting multiple admissions (as much as 18) while some did not get any. To bring out all these analyses, though, the school and student data from the Maharashtra RTE website had to be painstakingly downloaded, manually. Many DMers offered support to gather this data more easily.

datameet3

The team was quite inspired by the school map of the Karnataka Learning Partnership (klp.org.in/map) and wants to build such a comprehensive tool for themselves, with features to find schools within a specified distance, and help match students with schools. Unlike the Karnataka programme, there still is no MIS in place to facilitate the enrolment and selection process. Shravan suggested that it might be possible to use the codebase of KLP and adapt it for use in Ahmedabad. Hopefully, the D{M} folks will volunteer for the necessary support.

The RTE team also wants to build a tool to track the performance of enrolled students. They discussed about the potential privacy issues involved in this. It was suggested that the performance reporting to be published on the website could be at an appropriate level of aggregation which safeguards privacy and preserves discernible performance stats. The possibility of using ODK for volunteer led data collection was also discussed.
Getting together at the meetup opened up many possibilities for collaboration from the participants as a few of them came forward with suggestions and also extended their support to this cause.

Talk #2: Public Transport of Ahmedabad

Jayesh Gohel is not your everyday architect. He dropped out of his course at CEPT because he got too interested in code and soon enough he started enjoying making websites. Being an Amdavadi, he noticed the lack of infrastructure, both digital and non-digital in supporting the commuting that AMTS enabled in the city and so he decided to work on amtsinfo.in – the unofficial official support and information website for the Ahmedabad Municipal Transport Service.

Jaye

At the 2nd Datameet in Ahmedabad, Jayesh inspired the audience with his experiences with developing the website with the sole aim of solving the information problem related to the rather important and convenient network that AMTS is. Jayesh’s talk was simple and spoke about his personal motivations and learnings in the course of the development of this app. It also brought to the light the issues that plague the archaic systems that govern our modern lives, which can otherwise be so easily solved with the use of digital technology. However, ‘there’s hope if all of us take initiatives’, Jayesh said.

Open Transit Data for India

(Suvajit is a member of DataMeet’s Transportation working group, along with Srinivas Kodali, we are working on how to make more transit related data available.)

Mobility is one of the fundamental needs of humanity. And mobility with a shared mode of transport is undoubtedly the best from all quarters – socially, economically & environmentally. The key to effective shared mode of transport (termed as Public Transport) is “Information”. In India cities, lack of information has been cited as the primary reason for deterrence of Public Transport.

Transport Agencies are commissioning Intelligent Transport Systems (ITS) in various mode and capacity to make their system better and to meet the new transport challenges. Vehicle Tracking System, Electronic Ticketing Machines, Planning & Scheduling software are all engines of data creation. On the other side, advent of smart mobile devices in everyone’s hand is bringing in new opportunities to make people much more information reliant.

But the demand for transit data is remarkably low. The transit user and even transit data users like City Planners should demand for it.
The demand for Public Transport data in India should be for the following aspects:

A. Availability
To make operation and infrastructure data of Transport operators easily available as information to passengers in well defined order to plan their trip using available modes of Public Transport.

B. Interoperability
To make transit data provided by multiple agencies for different modes (bus, metro, rail) usable and make multi modal trip planning possible.

C. Usability
To publish transit oriented data in standard exchange format across agencies in regular frequencies to provide comprehensive, accurate and updated data for study, research, analysis, planning and system development.

D. Standardisation
To be a part of Passenger charter of Transport Operators to publish their data in standard format and frequency. This can also serve as a guideline for Transporter Operator while commissioning any system like Vehicle Tracking System, ITS, Passenger Information System, website etc.

What kind of Transit data is needed ?

  • Service Planning data

It will comprise of data on bus stops, stations, routes, geographic alignment, timetables, fare charts. With this dataset, general information on transit service can be easily gathered to plan a journey. Trip Planning mobile apps, portals etc can consume this data to provide ready and usable information for commuters.

  • Real time data

A commuter is driven by lot of anxieties when they depend on public transport mode. Some common queries; “When will the bus arrive ?”, “Where is my bus now?”, “Will I get a seat in the bus ?”, “Hope the bus has not deviated and not taking my bus stop.”.

Answer to all this queries can be attended via real time data like Estimated Time of Arrival (ETA), Position of the vehicle, Occupancy level , Alert and Diversion messages etc. Transport Operator equipped with Tracking systems should be able to provide these data.

  • Operational & Statistical Data

A Transport Operators operational data comprises of ticket sales, data of operation infrastructure and resources like Depots, Buses, Crew, Workshops etc. As operatore are tending towards digital mode of managing these data it also makes a good option to publish them at regular intervals.

A general commuter might not be interested in this data, but it will very useful for City Planners to analyse the trend of commute in the city and make informed decision. City transport infrastructure can be planned to orient it towards transit needs and demands.

The transport agency can benefit highly by demonstrating accountability and transparency. They can uplift their image as a committed service provider thereby gaining for passengers for their service.

So, together it will make a thriving landscape, if the data creators of Public Transport in India provide their data in Open which can be consumed by a larger set of people to build platforms, applications, solutions for transport study, analysis & planning across different section of users.

Open Transit Data is the tipping point for Smart Mobility in India.

That is why we have started putting our thoughts together and began writing an Open Transport Data Mainfesto.