Meeting 2018-01-19

Future Tasks

- Changes are noted by Bold
  • Frontend refactoring - Teyu
  • Take Multiple-Color-Pointmap as start point - Aaron
  • Stop words merge - Qiancheng
  • PointMap Parameter Configuration merge - Qiushi
  • "geoCell" & "getPoint" function - Qiushi
  • New middleware task finding for Qiancheng - Chen & Qiushi
  • Tableau Evaluation - Li Deng
  • Layers of Frontend - Yang Cao
  • Comparison with Oracle - Jiliang Ni

Notes: 
  • For people who need some sample data to play with, here is the link of 47000 records of Twitter data.
  • Data is `json` format:
The whole file is an array wrapped by []:
[
{tweet1},
{tweet2}
]

Each line/element (delimited by ",") is one tweet.
Example: "tweet1" - 
{
"create_at": "2016-01-01T00:00:00.000Z", //Timestamp the tweet was created.
"id": XXXXXXXXXXXX,
"text": "text content of this tweet",
...
"user": {sub-object, basic information of an user},
...
"place": {sub-object, location attributes when this tweet was published. One of them is `bounding_box` which contains 2 coordinates: [x1, y1] and [x2, y2] - diagonal corners of the rectangle bounding box where the tweet was published},
"geo_tag": {sub-object, this is an attribute added by us in order for our group-by aggregation UI, it includes "stateID", "stateName", "countID", "countyName", "cityID", "cityName"}
}
  • Here is an example query from our `Twittermap` application:
select `state` as `state`,coll_count(g) as `count`
from twitter t
where t.`create_at` >= datetime('2017-09-08T20:36:33.000Z') 
   and t.`create_at` < datetime('2017-09-10T20:36:33.000Z') 
   and t.`geo_tag`.`stateID` in [ 37,51,24,11,10,34,42,9,44,48,35,4,40,6,20,32,8,49,12,22,28,1,13,45,5,47,21,29,54,17,18,39,19,55,26,27,31,56,41,46,16,30,53,38,25,36,50,33,23,2 ]
group by t.geo_tag.stateID

of course, the `stateID` might be replaced by `countyID` or `cityID`
  • The data is the same with our website's `Quick Start` local demo's data.
  • If someone would play with our local demo, please use this `Quick Start` to get your hands dirty.
CrimeData

Comments

Post a Comment

Popular posts from this blog

Meeting 2018-02-02

Future Tasks - Changes are noted by  Bold Reconstruct AsterixDB install guide  -  Qiushi Maintenance jobs - Qiushi Frontend refactoring - Teyu Take Multiple-Color-Pointmap as start point - Aaron "geoCell" & "getPoint" function - Qiushi Wrap the conf file parameters loaded by Twittermap  - Qiancheng Tableau Evaluation - larger data set - Li Deng Goes down to performance issue, pulling data from AsteridDB, like Nik's work - Yang Cao Comparison with Oracle - more experiments - Jiliang Ni Achievements Solve the AsterixDB installation problem   -  Taewoo Run Oracle and import Tweets data into Oracle - Jiliang Ni Load different layers of data into UI - Yang Cao Show 2 colors of points on map - Aaron Notes Rename "Point Map" to "Pin Map" - suggestion Build specific keywords (e.g. "HIV") TwitterMap instances - Colaboraters
Read more

TODOs - Mar 16 2018

Frontend (Twittermap)  - Right sidebar is hidden by scrollbar  - Pinmap left coner tool box needs to be updated or removed  - Popup Tweet content by talking to our own DB - Qiancheng is working  - Integrate multi-layered frontend feature  - Enabling time series bar data caching Bugs  - First time search for a new keyword, the slicing queries might be stuck as the second mini-query.  - "Future timeout" exception (where and when is not known now, sometimes when starting the TM) Features  - New 47000 size Sample Data (with user.pic_url attribute) is needed for Quick-Start guide environment  - HIVmap should be merged into our Github repository in some way.  - Supporting "Geocell group by" queries with arbitrary scale (currently is fixed) as parameter in Cloudberry  - A Guard monitoring our whole system's status  - Randomly generate coordinate for each tweet which does not have coordinate value during geotagging preprocessing Other
Read more

Meeting 2018-01-12

Independent Tasks for new members (1) Tableau Evaluation     - Related to this blog ;     - Tableau website: https://www.tableau.com/     - Instructions:         - Install the software         - Put some sample data into it         - Play around with its features and UIs, get some sense of its power         - Look deeper into it and try to translate what kinds of queries can Tableau answer and evaluate how fast it can reply (2) Layers of Frontend     - Take Google Maps as an example, it allows realtime add or remove different layers of dataset: e.g. Terrain, Traffic, Bicycling routes, Satellite, etc.     - Explore how to implement a frontend with such features of realtime adding and removing layers of different datasets     - Instructions:         - Find some techniques or libraries to do this         - Do a toy demo to get some sense of how it works and how to implement this         - Then we will see how to implement this based on our current codebase (3) Compariso
Read more