UCC-Organism/uccorg-backend

语言: CoffeeScript

git: https://github.com/UCC-Organism/uccorg-backend

后端处理夜间数据转储并将其推送到前端
Backend for processing the nightly data dump and pushing it to the frontend
README.md (中文)

UCC有机体后端

ci

UCC有机体的后端

信息

服务器使用coffee uccorg-backend.coffee configfile.json运行,其中configfile.json包含服务器的实际配置。 根据配置,运行方式如下:

  • 生产数据准备服务器(windows),负责从ucc / webuntis / calendar / ...获取数据,匿名化它们,然后将它们发送到api服务器
  • 生产api服务器(debian on macmini),它从数据准备服务器获取匿名数据,通过api使它们可用,并使用Bayeux协议发出事件
  • 用于后端的开发服务器,它使用实际数据转储而不是与外部服务通信
  • 自动化测试,使用travis自动运行,使用示例数据转储,并模拟运行速度非常快的时间。
  • 用于前端的开发服务器, - 运行样本数据转储并模拟时间, - 能够获得事件而无需等待实际活动

组态

config.json.sample中列出了所有配置选项。另请参阅test.json了解实际配置,这个配置的内容也是前端开发服务器的一个不错的选择, - 只需删除“outfile”,并减少时间速度因子“xTime” - 这说明速度快了多少应该运行模拟时钟。

根据需要推送到机器人/屏幕的客户端配置在client-config / default.js中

API

api提供JSON对象,可通过http获得,启用JSONP和CORS。端点是:

  • /(agent | location | event)/ $ ID返回有关特定实体的信息
  • / now /(agent | location)/ $ ID返回具有当前状态的对象,注意:这随时间变化
  • / current-state返回活动代理+活动
  • / client-config返回客户端配置

老api: - /(agents | locations | events)返回id列表。注意:这是用于开发/探索,并且预计不会在生产中使用,因为某些事件/代理将在运行中创建并且可能不在列表中 - /(teacher | group | activity)/ $ ID返回有关特定实体的信息 - /(teachers | groups | activities)返回id列表

通过faye(http://faye.jcoglan.com/)发生的事件被推送/事件,即。 (new Faye.Client('http:// localhost:8080 /')).subscribe('/ events',function(msg){...})

统一代理人调度笔记

数据模式: - 代理人:(教师或学生团体成员)   - ID    - 善良:老师|学生|研究员|看门人|巴士|火车| ...    - ?性别:0/1(适合教师/学生)    - ?ids组(学生)    - ?程序:    - ?年龄(学生)    - ?结束(学生)    - ?名称(火车/巴士)    - ?原产地(火车/公共汽车) - 活动 - 活动,火车到达等   - 时间    - TODO类型:活动开始,活动结束,巴士到达,火车到达,    - 代理商    - ?描述:活动主题等   - ?位置    - TODO?活动 - 链接到ucc-activity进行调试 - 地点   - ID   - 名称    - 容量

生产服务器设置

对于生产系统,我们选择在现有的mac上使用虚拟Linux服务器。

api-server服务器的要求:

  • 停电时自动启动等
  • 运行linux(其他操作系统也应该工作,但我不会给它任何支持)
  • 在端口8080上运行http服务时,它将在内部网络上以http://10.251.26.11:8080的形式提供
  • 某种方式重新启动它,并远程访问它

直接选项是

  • 在UCC网络上设置专用服务器
  • 在mac上运行虚拟服务器,驱动其中一个显示器

由于已经在mac上配置了虚拟服务器,并且假设它在生产中保持运行,我们选择后者。

在linux上的服务器设置,作为用户uccorganism与home / home / uccorganism。假设git / nc通过包管理器安装,node / npm / coffee安装在/ usr / local / bin中。服务器的安装完成:

cd
git clone https://github.com/UCC-Organism/uccorg-backend.git
cd uccorg-backend
npm install
(crontab -l; echo @reboot /home/uccorganism/uccorg-backend/apiserver-start.sh) | sort | uniq | crontab -

然后api-server应该在重启后运行

关于外部数据源的假设

数据源

  • webuntis通过API密钥访问
  • / api / [locations | subject | lessons | groups | teachers | departments]返回id的json数组
  • / api / locations / $ ID返回json对象:untis_id,capacity,longname
  • / api / subjects / $ ID返回json对象:untis_id,name,longname,alias
  • / api / lessons / $ ID返回json对象:untis_id,start,end,subject,teachers,groups,locations,course
  • / api / groups / $ ID返回json对象:untis_id,name,alias,schoolyear,longname,department 别名必须与mssql数据库中的“Holdnavn”匹配
  • / api / teachers / $ ID返回json对象:untis_id,name,forename,longname,departments 该名称必须与mssql-database中的“Initialer”匹配
  • / api / departments / $ ID返回json对象:untis_id,name,longname
  • ucc mssql数据库 - 可通过服务器上的存储过程访问:
  • GetStuderendeHoldCampusNord:团队名称(= groups.alias),研究编号
  • GetAnsatteHoldCampusNord:团队名称,缩写
  • GetAnsatteCampusNord:Initialer,Afdeling,Køn webuntis中的每位教师都必须有一个条目,其中Initialer与teacher.name相同
  • GetStuderendeCampusNord:学号,分部,性别(0/1 - > Agent.gender),生日(DDMMYY - > Agent.gender)
  • GetHoldCampusNord:部门,团队名称,描述,StartDate,EndDato 员工/学生的每个团队名称应该有一个团队,还有一个团队与webuntis的每个groups.alias匹配
  • google-calendar返回rrule可解析的数据,SUMMARY作为事件的标题
  • rejseplanen-api http://xmlopen.rejseplanen.dk/bin/rest.exe/arrivalBoard?id = 8600683&date = ...以表格<Arrival name =“...”type =“...返回到达时间表“date =”...“origin =”......“>。

状态

发布日志

2015年1月至4月

  • 整体
  • √所有代理人类型的统一代表,例如研究人员,教育工作者,踏板,厨房工作人员等,以与学生相同的方式表示:关联团体,在房屋之间移动等。
  • 代理人的偶然行为,例如教学,上厕所,午餐等之间的休息。
  • √globale模式,如:日间周期,支付su等
  • 配置随机行为和事件的机会
  • 在公司成立之前,后端对A&K的愿望和前端开发正在进行中
  • √即使外部数据源出现故障,系统也会继续运行
  • √操作配置的说明, - 我们应该设置一个单独的Linux服务器,还是与运行屏幕的Mac并行运行
  • √不需要(最终配置和设置linux服务器)
  • √积极与前端开发进行持续沟通,确保后端符合前端的愿望和期望
  • √记录外部数据源的期望和要求
  • 第16周
  • 强度等级0-1 +随机的全局事件
  • 延迟时间(在Mac上弯曲的手柄)
  • 第15周
  • 不区分大小写+修剪日历事件
  • README中使用过的数据源的文档
  • 两种色彩范围的随机颜色+专业的每个代理商。
  • 第14周
  • 错误修正:无休止的漫游
  • 决定/文件制作设置
  • 可配置的类缩短以创建中断
  • 可配置的配置+日历+静态数据,用于系统配置更改时的可重复单元测试
  • 为管理员,门卫,厨房和咖啡馆/食堂配置活动
  • 修复伪随机 - 由于imul而导致的错误
  • (在Linux上运行的前端)
  • 第13周
  • 随机行为 只有代理商出现随机事件 随机事件发射 - 背景和恢复其他事件 生成随机事件 可配置的随机行为 - 午餐,厕所休息,病假,pauser mellem undervisning等。 随机哈希均匀分布 样本随机配置传递到系统中 来自schedule + isScheduled的事件的前缀 重构旁边是一个函数
  • 更友好的url事件(使用“_”代替“”)
  • / now中的event-id
  • 事件也带来了可能的结束时间
  • 实现/下一个api端点
  • 第12周
  • 为代理添加随机漫游/离开状态 - 对于校园中的代理没有特别做任何事情的随机事件所需 可在data / general-settings.json中配置
  • 服务器错误消息,如果在json中出错
  • 第10周
  • 全球国家 - 天周期等通过代理人 - 即。 / agent / time-of-day day cycle - grant,su等可配置
  • 版本控制中的apiserver-script
  • 支持远程停止/重新启动API服务器
  • API服务器中的日历数据检索
  • internal:使用哈希函数保留事件ID的顺序,以避免由于同时更改事件顺序而导致的测试错误。
  • 第9周
  • 更改数据中配置文件的结构/使其更易于编辑:data / behavior.js拆分为locations.json,agents.json,calendar.js,og behaviour.js
  • 开始将日历检索从数据处理转移到API服务器
  • 第8周
  • 如果我们最近没有从数据服务器获得任何更新,请重复旧数据
  • 第7周
  • 通过将代理分配到位置来处理多个位置
  • 从日历创建活动和代理
  • data / behaviour.js中日历行为的配置
  • 第6周
  • 在状态中包含警告,并将警告从Windows服务器传播到api-server
  • 公共汽车/火车事件作为统一事件而不是单独到达
  • 联系UCC关于SSMLDATA-server已关闭
  • 一些重构/死代码消除
  • 第5周
  • webservice获取代理和位置的当前状态,替换old / now /
  • 发出的事件来自新的统一api
  • 新的测试数据(但禁用了休息测试)
  • 第4周
  • 起草新的api:/ events,/ agents,/ locations
  • agent +事件统一静态数据
  • 前端开发的临时代理

在2014年的发展之间

  • 丢失数据时的虚拟保持
  • 数据的解决方法,其中几个组具有相同的untis_did
  • 当缺少教师,位置或主题时,创建具有“错误:缺失”的虚拟数据
  • 学生年龄
  • 忽略webuntis的坏ssl证书 - 因为它们/(是?)错误

里程碑3 - 运行至2014年1月20日

  • 配置mac-mini自动启动api-server
  • 仪表板
  • 从远程日历(火车时刻表等)获取数据
  • 修复时区错误(测试夏令时处理)
  • 不再隧道传输数据,而是在防火墙打开时通过端口8080将数据直接发送到macmini。
  • 更新Windows服务器上的配置,发送当前日期,而不是将来一个月进行测试。
  • prepare-server:支持转储到文件以进行开发
  • 仪表板:在事件发生时显示事件
  • 仪表板骨架
  • 添加api以获取所有教师/团体/地点/活动的ID

里程碑2 - 运行至2013年12月29日

  • Windows服务器配置为每晚1点钟提取数据,并将它们发送到mac mini。
  • 添加api以获取当前/下一个/上一个活动给定位置,教师或小组
  • 更新REST-api
  • 将配置移动到配置文件中
  • 从ucc / webuntis-data为apiserver生成数据文件
  • 匿名学生
  • 暂时通过ssl.solsort.com转发数据,因为从ssmldata到macmini的端口8080似乎没有打开。
  • 将数据从ssms data-server发送到mac mini

里程碑1 - 运行至2013年12月22日

  • 从sqlserver获取数据
  • 从webuntis获取数据
  • 得到mac mini到UCCs serverpark
  • 通过王菲的现场活动
  • 通过api获取数据
  • 有权访问配置的ssmldata-server
  • 用于自动测试的虚拟数据集
  • mac mini上的自动测试,具有快速开发时间
  • 不推荐使用solsort.com运行基于webuntis的webservice进行开发

数据问题

  • webuntis上的一些老师从mssql中丢失(因此缺少性别非关键)
  • 缺少个别课程和团队之间的信息,只有年末级别的信息和每门课程后的年度课程
  • 信息通过mssql缺少以下组:fss12b,fss11A,fss11B,fsf10a,fss10,fss10b,fss12a,norF14.1,norF14.2,norF14.3,nore12.1和“SPL M5 - F13A和F13B”
  • 活动对于特定时间的团体/位置不一定是唯一的,这会稍微扰乱当前/下一个活动api,这只会返回下一个/上一个单一的
  • 处所的命名可能不一致

主要

请参阅config.json-sample和test.json中的示例文件。

config = {}
if require.main == module then do ->
  try
    configfile = process.argv[2]
    configfile = "config" if !configfile
    configfile += ".json" if configfile.slice(-5) != ".json"
    config = JSON.parse (require "fs").readFileSync configfile, "utf8"
  catch e
    console.log "reading config #{configfile}:", e
    process.exit 1

  config.icalCacheFile ?= "cached-calendar.ical"
  config.configData ?= "data"

  process.nextTick ->
    if config.prepare then dataPreparationServer() else apiServer()

常见的东西

关于

exports.about =
  title: "UCC Organism Backend"
  author: "Rasmus Erik Voel Jensen"
  description: "Backend for the UCC-organism"
  owner: "UCC-Organism"
  name: "uccorg-backend"
  dependencies:
    solapp: "*"
  package:
    scripts:
      test: "solapp build; rm -f test.out ; ./node_modules/coffee-script/bin/coffee uccorg-backend.coffee test ; diff test.out test.expected"
    dependencies:
      async: "0.2.9"
      "coffee-script": "1.6.3"
      express: "3.4.6"
      faye: "1.0.1"
      mssql: "0.4.1"
      request: "2.30.0"
      rrule: "2.0.0"
      "js-beautify": "1.5.4"
      "jshint": "2.6.0"

依赖

assert = require "assert"
fs = require "fs"
express = require "express"
http = require "http"
faye = require "faye"
async = require "async"
mssql = require "mssql"
request = require "request"

实用功能

Math.imul

Math.imul ?= (a,b)->
  _ah = (a >>> 16) & 0xffff
  _al = a & 0xffff
  _bh = (b >>> 16) & 0xffff
  _bl = b & 0xffff
  (_al * _bl) + (((_ah * _bl + _al * _bh) << 16) >>> 0)|0

甚至型

eventType = (eventObject) -> (eventObject.description || "undefined").split(" ")[0]

djb2哈希

hash = (str) ->
  result = 5381
  for i in [1..str.length-1]
    result = 33 * result + str.charCodeAt(i) &0x7fffffff
  result

uniqueHash

uniqueHashes = {}
uniqueHash = (s) ->
  h = hash(s)
  s2 = uniqueHashes[h]
  while s2 && s2 != s
    h += 1
    s2 = uniqueHashes[h]
  uniqueHashes[h] = s
  h

pseudoRandom实用程序函数

prand = (i) ->
  i ?= 0
  next = -> 
    i = Math.imul(1103515245, i) + 12345 & 0x7fffffff
  return {
    next: -> next(); (i & 0x3fffffff) / 0x40000000
    nextN: (n) -> next(); n * (i & 0x3fffffff) / 0x40000000 | 0
  }

伪随机

seed = prand(0)
pseudoRandom = -> seed.next()

唯一身份

uniqueId = do ->
  prevId = 0
  return -> prevId += 1

getDateTime

获取当前时间为yyyy-mm-ddThh:mm:ss(本地时区, - 或运行test / dev时的模拟值)

timeoffset = 0
timecorrection = 0
getDateTime = -> (new Date(Date.now() + timecorrection - (new Date).getTimezoneOffset() * 60 * 1000 + timeoffset)).toISOString().slice(0,-1)

如果非常错误,请纠正时间

从http获取时间。请注意,这是非常不精确的,服务器设置应该通过NTP找到正确的时间。不幸的是,我们似乎是在一个限制性的防火墙后面,所以如果我们可以看到本地时钟漂移的方式错误,我们会尝试解决它。

if require.main == module then setImmediate ->
  (http.get "http://www.google.com", (o) ->
    requestTime = Date.now()
    remoteTime = new Date(o?.headers?.date)
    return warn "error getting date from www.google.com for clock sanity check" if String(remoteTime) == "Invalid Date"

这是非常不精确的,请求时间发生在远程时间+网络延迟 我们不知道网络延迟(和http请求包括握手, 所以也很难估计)。

无论如何,remoteTime具有精确的秒数和网络延迟 不应该是很多秒,所以偏移应该是精确的 几秒钟, - 如果我们能观察漂移,我们只会应用它 超过半分钟:(

    timeOffset = (+remoteTime) - requestTime

    if Math.abs(timeOffset) > 30*1000
      warn "servertime is more the 30s off (#{timeOffset / 1000}), compared to google webservers + network latency, - local clock is probably very drifting, and thus we apply imprecise corrections"
      timecorrection = timeOffset
  ).on("error", (-> warn "error connecting to www.google.com for clock sanity check"))

睡觉

设置超时的语法更舒服:sleep #seconds, - > ...

sleep = (t, fn) -> setTimeout fn, t*1000

binarySearchFn

binSearchFn = (arr, fn) ->
  start = 0
  end = arr.length
  while start < end
    mid = start + end >> 1
    if fn(arr[mid]) < 0
      start = mid + 1
    else
      end = mid
  return start

if typeof isTesting == "boolean" && isTesting then do ->
  arr = [0,1,2,3,4,5]
  assert.equal 2, binSearchFn arr, (a) -> a - 2
  assert.equal 3, binSearchFn arr, (a) -> a - 2.1

sendUpdate

sendUpdate = (data, callback) ->
  datastr = JSON.stringify data

像uncii一样逃脱unicode

  datastr = datastr.replace /[^\x00-\x7f]/g, (c) ->
    "\\u" + (0x10000 + c.charCodeAt(0)).toString(16).slice(1)
  opts =
    hostname: config.prepare.dest.host
    path: config.prepare.dest.path || "/uccorg-update"
    port: config.prepare.dest.port || undefined
    method: "post"
    headers:
      "Content-Type": "application/json"
      "Content-Length": datastr.length
  if !(config.prepare.dest.protocol in ["http", "https"])
    throw "config error: prepare.dest.protocol neither 'http' nor 'https'" 
  req = (require config.prepare.dest.protocol).request opts, callback
  console.log "sending data: #{datastr.length} bytes"
  req.write datastr
  req.end()

状态

status =
  bootTime: getDateTime()
  warnings: {}
warn = (msg) ->
  status.warnings[msg] = getDateTime()

generalSettings =
  try (require "./#{config.configData}/general-settings.json")
  catch e
    warn "Error in configuration in #{config.configData}/ " + e
    {}

generalSettings.minRoam ?= 15
generalSettings.maxRoam ?= 30

日历

数据准备 - 在SSMLDATA服务器上运行处理/提取

dataPreparationServer = ->

SQL Server数据源

  getSqlServerData = (done) ->
    if config.prepare.mssqlDump
      try
        result = JSON.parse fs.readFileSync config.prepare.mssqlDump
        return done? result
      catch e
        console.log "Loading mssql dump:",  e

    entries = ["Hold", "Studerende", "Ansatte", "AnsatteHold", "StuderendeHold"]
    result = {}
    return done?(result) if not config.prepare.mssql

    handleNext = ->
      if entries.length == 0
        if config.prepare.mssqlDump
          fs.writeFileSync config.prepare.mssqlDump, (JSON.stringify result, null, 2)
        return done?(result)
      current = entries.pop()
      console.log "mssql", current
      req = con.request()
      req.execute "Get#{current}CampusNord", (err, reqset) ->
        throw err if err
        result[current] = reqset
        handleNext()

    con = new mssql.Connection config.prepare.mssql, (err) ->
      throw err if err
      handleNext()

Webuntis数据源

我们还不知道是否应该使用webuntis api,或者从ucc获取单个数据转储 如果需要,从old-backend-code.js中提取代码

  getWebUntisData = (callback) ->
    if config.prepare.webuntisDump
      try
        result = JSON.parse fs.readFileSync config.prepare.webuntisDump
        return callback?(result)
      catch e
        warn "error loading webunits dump"
        console.log "Loading webuntis dump:", e

    do ->
      apikey = config.prepare.webuntis
      untisCall = 0

webuntis - 用于调用webuntis api的函数

      webuntis = (name, cb) ->
        if ((++untisCall) % 100) == 0
          console.log "webuntis api call ##{untisCall}: #{name}"
        url = "https://api.webuntis.dk/api/" + name + "?api_key=" + apikey
        request {
            url: url
            rejectUnauthorized: false
          }, (err, result, content) ->
            warn "webuntis request error #{name}" if err
            return cb err if err
            cb null, JSON.parse content

提取数据,从webuntis下载所需的数据

      extractData = (dataDone) ->
        result =
          locations: {}
          subjects: {}
          lessons: {}
          groups: {}
          teachers: {}
          departments: {}

        async.eachSeries (Object.keys result), ((datatype, cb) ->
          console.log "getting #{datatype} from webuntis"
          webuntis datatype, (err, data) ->
            cb err if err
            async.eachSeries data, ((obj, cb) ->
              id = obj.untis_id
              webuntis "#{datatype}/#{id}", (err, data) ->
                result[datatype][id] = data
                cb err
            ), (err) -> cb err
        ), (err) ->
          dataDone err, result

      extractData (err, data) ->
        warn "extractData error" if err
        throw err if err
        if config.prepare.webuntisDump
          fs.writeFileSync config.prepare.webuntisDump, (JSON.stringify data, null, 2)
        callback?(data)

转换事件/ api-server的数据

  processData = (webuntis, sqlserver, callback) ->
    startTime = config.prepare.startDate || 0
    if typeof startTime == "number"
      startTime = (new Date(+(new Date()) + startTime * 24*60*60*1000)).toISOString()
    else
      startTime = (new Date(startTime)).toISOString()
    endTime = (new Date(+(new Date(startTime)) + (config.prepare.timespan || 1) * 24*60*60*1000)).toISOString()
    console.log "Extracting data from #{startTime} to #{endTime}"

存储库中的文件包含用于测试的示例数据。

对于每种数据,都有从id到单个对象的映射

输出描述

  • 活动:身份证,开始/结束,教师,地点,主题,群体
  • 组:id,组名,程序,学生(id,性别)
  • 老师:身份,性别,计划
  • 位置:id,名称,长名称,容量
  • calendarEvents:开始,结束,标题,描述

初始化

    result =
      locations: {}
      activities: {}
      groups: {}
      teachers: {}

地点

    for _, location of webuntis.locations
      result.locations[location.name] =
        id: location.name
        name: location.longname
        capacity: location.capacity

addTeacher

    teachers = {}
    for obj in sqlserver.Ansatte[0]
      teachers[obj.Initialer] = obj

    addTeacher = (obj) ->
      id = obj.untis_id
      name = obj.name
      warn "missing teacher in mssql" if !teachers[name]
      result.teachers[id] =
        id: id
        gender: teachers[name]?["Køn"]
        programme: teachers[name]?["Afdeling"]
        programmeDesc: do ->
          id = obj.departments[0]
          dept = webuntis.departments[id]
          "#{dept?.name} - #{dept?.longname}"

addGroup(和学生)

    students = {}
    studentId = 0
    studentIds = {}
    getStudentId = (studienummer) -> studentIds[studienummer]

    for obj in sqlserver.Studerende[0]
      studentIds[obj.Studienummer] = ++studentId
      studentObject =
        id: getStudentId obj.Studienummer
        gender: obj["Køn"]

从生日算起年龄

      today = new Date()
      birthday = obj["Fødselsdag"]
      if birthday

        birthyear = parseInt(birthday.slice(4,6), 10)

修复两位数的年份问题, 即。 “12”可能是1912年和2012年 假设21世纪,如果它在今天之前。

        birthyear += 100 if birthyear < (today.getYear() - 100)

        birthmonth = parseInt(birthday.slice(2,4), 10)
        birthdate = parseInt(birthday.slice(0,2), 10)

        age = today.getYear() - birthyear
        age -= 1 if new Date(today.getYear(), birthmonth - 1, birthdate) > today

        studentObject.age = age

      studentObject.end = obj.Forventet_slutdato if obj.Forventet_slutdato

      students[getStudentId obj.Studienummer] = studentObject

    groups = {}
    for obj in sqlserver.Hold[0]
      groups[obj.Holdnavn] =
        name: obj.Holdnavn
        department: obj.Afdeling
        start: obj.StartDato
        end: obj.SlutDato
        students: []
    for obj in sqlserver.StuderendeHold[0]
      if !groups[obj.Holdnavn]
        console.log 'Data error: Hold missing for StuderendeHold', obj
        warn 'Data error: Hold missing for StuderendeHold ' + obj.Holdnavn

假人抱

        groups[obj.Holdnavn] =
          name: obj.Holdnavn
          department: '219405'
          start: '01-01-2014'
          end: '01-01-2060'
          students: []
      groups[obj.Holdnavn].students.push students[getStudentId obj.Studienummer]

    addGroup = (obj) ->
      return obj.untis_id if result.groups[obj.untis_id]

Buggy数据:有时同一组有几个untis_id,这就是为什么我们用json.parse / stringify做一个深层复制的原因

      grp = result.groups[obj.untis_id] = JSON.parse JSON.stringify groups[obj.alias] || {}
      grp.id = obj.untis_id
      grp.group = obj.name
      dept = webuntis.departments[obj.department]
      warn "Department missing: #{obj.department}" if !dept and obj.department
      grp.programme = "#{dept?.name} - #{dept?.longname}"
      grp.id

添加所有组组和教师

确保日历活动需要时可以使用它们

    do ->
      for _, teacher of webuntis.teachers
        addTeacher teacher

      for _, group of webuntis.groups
        addGroup group

处理活动

    for _, activity of webuntis.lessons
      if startTime < activity.end && activity.start < endTime && activity.end
        result.activities[activity.untis_id] =
          id: activity.untis_id
          start: activity.start
          end: activity.end
          teachers: activity.teachers.map (untis_id) ->
            addTeacher webuntis.teachers[untis_id] || {untis_id: untis_id, name: "error:missing", departments: ["error:missing"] }
            untis_id
          locations: activity.locations.map (loc) -> webuntis.locations[loc]?.name || "error:missing"
          subject: activity.subjects.map((subj) -> webuntis.subjects[subj]?.longname || "error:missing").join(" ")
          groups: activity.groups.map (untis_id) ->
            addGroup webuntis.groups[untis_id] || {untis_id: untis_id}

DONE

    callback result

执行

  getWebUntisData (data1) ->
    getSqlServerData (data2) ->
      processData data1, data2, (result) ->
        result.status = status
        if config.prepare.dest.dump
          fs.writeFile config.prepare.dest.dump, JSON.stringify(result, null, 2)
        sendUpdate result, (err, data) ->
          if err
            console.log 'sendUpdate error:', err
            warn 'sendUpdate error'
          console.log "submitted to api-server"
          process.exit 0

事件/ API服务器

apiServer = ->
  data = undefined
  activitiesBy =
    group: {}
    location: {}
    teacher: {}
  eventsByAgent = {}

的CalendarData

  calendarData = (done) ->
    icaldata = []

    request config.icalUrl || "http://no-ical-url-in-config/", (err, result, ical) ->
      if err
        warn 'Error getting calendar data ' + config.icalUrl
        ical = fs.readFileSync config.icalCacheFile, "utf8"
      else
        fs.writeFile config.icalCacheFile, ical

      return done() if !ical

      icaldata = []
      !ical.replace /BEGIN:VEVENT([\s\S]*?)END:VEVENT/g, (_,e) ->
        props = e.split(/\r\n/).filter((x) -> x != "")
        event = {}
        for prop in props
          pos = prop.indexOf ":"
          pos = Math.min(pos, prop.indexOf ";") if prop.indexOf(";") != -1
          event[prop.slice(0,pos)] = prop.slice(pos+1)
        icaldata.push event

      calId = 0
      result = []
      startTime = getDateTime()
      endTime = +(new Date(getDateTime())) + 7 * 24 * 60 * 60 * 1000
      result.calendarEvents = []

      handleEvent = (dtstart, event) ->
        activity =
          id: "cal#{++calId}"
          start: dtstart.toISOString()
          end: new Date(+dtstart + (+iCalDate(event.DTEND) - +iCalDate(event.DTSTART))).toISOString()
          type: String(event.SUMMARY).trim().toLowerCase()
        result.push activity

      iCalDate = (t) ->
        d = t.replace /.*:/, ""

警告:这里我们假设我们在欧洲/哥本哈根时区

        d = new Date(+d.slice(0,4), +d.slice(4,6) - 1, + d.slice(6,8), +d.slice(9,11), +d.slice(11,13), +d.slice(13,15), 0)
        d = new Date(+d - d.getTimezoneOffset() * 60 * 1000)
        if (t.slice(0, 23) == "TZID=Europe/Copenhagen:") || (t.slice(0,11) == "VALUE=DATE:")
          d
        else if t.slice(-1) == "Z"
          d = new Date(+d - d.getTimezoneOffset() * 60 * 1000)
        else
          warn "timezone bug in calendar data " + t + " " + d
          console.log "timezone bug in calendar data", t, d
        d

      if icaldata then for event in icaldata
        if event.RRULE
          RRule = (require "rrule").RRule
          opts = RRule.parseString event.RRULE
          opts.dtstart = iCalDate event.DTSTART
          rule = new RRule(opts)
          occurences = rule.between(new Date(startTime), new Date(endTime), true)
          for occurence in occurences
            handleEvent occurence, event
        else if startTime <= iCalDate(event.DTSTART).toISOString() < endTime
          handleEvent iCalDate(event.DTSTART), event

      data.calendar = result if data?
      done result

计算每小时一次的时间偏移,一周倒回时钟,如果自上次数据更新后超过八天

  updateTimeOffset = ->
    lastUpdate = new Date(status.lastDataUpdate)
    if 8 < (new Date() - lastUpdate + timeoffset) / 1000 / 60 / 60 / 24
      timeoffset -= 7 * 24 * 60 * 60 * 1000
      enrichData()
  setInterval updateTimeOffset, 1000 * 60

处理数据

每天从UCC推送到服务器。

  handleUCCData = (input, done) ->
    console.log "handling data update from ucc-server"
    fs.writeFile config.apiserver.cachefile, JSON.stringify(input), ->
      data = input
      if input.status and input.status.warnings
        for key, val of input.status.warnings
          status.warnings[key] = "data " + val
      calendarData enrichData
      console.log "data replaced with new data from ucc-server"
      done()

  enrichData = -> #{{{3

    activitiesBy =
      group: {}
      location: {}
      teacher: {}

包含按组/位置/教师排序的活动的表格

    for _, activity of data.activities
      for kind, collection of activitiesBy
        for elem in activity["#{kind}s"]
          collection[elem] ?= []
          collection[elem].push activity
    for _, collection of activitiesBy
      for _, arr of collection
        arr.sort (a, b) -> a.end.localeCompare b.end

添加代理+事件

代理/事件数据结构

    assignAgentColors = (agent) ->
      seed = prand(hash(agent.id))

      randomComponent = (a, b) ->
         a = parseInt a, 16
         b = parseInt b, 16
         n = 0x100 + a + seed.next() * (b - a) | 0
         n.toString(16).slice(1)

      randomColor = (c1, c2) ->
        r = randomComponent(c1.slice(1,3),c2.slice(1,3))
        g = randomComponent(c1.slice(3,5),c2.slice(3,5))
        b = randomComponent(c1.slice(5,7),c2.slice(5,7))
        "#" + r + g + b

      colors = generalSettings.agentColors?[agent.programme]
      colors = generalSettings.agentColors?.default if !colors
      return if !(colors?.color1min and colors.color2min and colors.color1max and colors.color2max)

      agent.color1 = randomColor(colors.color1min, colors.color1max)
      agent.color2 = randomColor(colors.color2min, colors.color2max)

    data.agents = {} #{{{3
    for _, teacher of data.teachers
      id = "teacher" + teacher.id
      data.agents[id] = agent = {}
      agent.kind = "teacher"
      agent.gender = teacher.gender
      agent.programme = teacher.programmeDesc
      agent.id = id
      assignAgentColors(agent)

    data.agents.JamesBond =
      kind: "yes"
      gender: 1
      license: "kill -9"
      id: "007"
      description: "undercover testagent"

    for groupId, group of data.groups
      continue if not group.students
      for student in group.students
        id = "student" + student.id
        data.agents[id] = agent = data.agents[id] || {}
        if agent.programme && group.programme != agent.programme
          console.log "warning: student in several programmes, ignoring", id, group.programme, agent.programme
          warn "warning: student in several programmes, ignoring " + id  + " " + group.programme + " " + agent.programme
        agent.kind = "student"
        agent.programme = group.programme
        agent.groups ?= []
        agent.groups.push groupId
        agent.age = student.age
        agent.gender = student.gender
        agent.end = student.end
        agent.id = id
        assignAgentColors(agent)

    data.events = {} # {{{3

    addEvent = (agents, location, time, description, misc) ->
      if !agents || !agents.length
        return

id = time +''+ hash(“”+ agents + location + description)+''+ uniqueId()

      id = time + '_' + uniqueHash("" + agents + location + description) + '_'+ ((description || "").split " ")[0]
      data.events[id] =
        id: id
        location: location || undefined
        description: description
        time: time
        agents: agents
      if misc
        for key, val of misc
          data.events[id][key] ?= val
      id

    addEvents = (agents, locations, time, description, misc) ->
      len = locations.length
      if len > 1

将代理分配到事件的位置

        for i in [0..len-1] by 1
          addEvent (agents[j] for j in [i..agents.length-1] by len),
            locations[i], time, description, misc
      else
        addEvent agents, locations[0], time, description, misc


    seed = prand(0)
    for _, activity of data.activities
      agents = []
      for teacherId in activity.teachers
        agents.push "teacher" + teacherId
      for groupId in activity.groups
        for student in data.groups[groupId].students || []
          agents.push "student" + student.id

      addEvents agents, activity.locations, activity.start, "scheduled " + activity.subject, { likelyEndTime: activity.end}
      minBreak = config.minBreak || 1 / 59
      maxBreak = config.maxBreak || 1 / 59
      breakRandom = prand(hash("" + activity.start + activity.locations + activity.agents))
      breakTime = 1000*60*(minBreak+breakRandom.next() * (maxBreak - minBreak))
      endTime = (new Date(new Date(activity.end.slice(0,19)+'Z') - breakTime)).toISOString().slice(0,19)
      addEvent agents, undefined, endTime, "roaming", {autoLeave: leaveTime(endTime + agents)}

代理人代理人  addEvent [agent],undefined,(new Date(new Date(activity.end.slice(0,19)+'Z') - ( - (generalSettings.minRoam +(generalSettings.maxRoam - generalSettings.minRoam)* pseudoRandom() )| 0)* 60 * 1000))。toISOString()。slice(0,19),“away”

    randomEvents = (start, end, events) -> #{{{3
      for o in events
        o.minDuration ?= 0.1
        o.maxDuration ?= 10
        try
          for agentId, agent of data.agents
            if agent.kind in o.agentTypes
              time = +(new Date(start))
              end = +(new Date(end))
              random = prand(hash(agentId + start + o.description))
              time += 2 * random.next() * 60 * 60 * 1000 / o.frequencyPerHour
              while time < end
                location = o.locations[random.nextN o.locations.length]
                startEvent = addEvent [agentId], location, (new Date(time)).toISOString(), "random " + o.description, {during: o.during}
                endTime = time + 1000*60* (o.minDuration  + (o.maxDuration - o.minDuration) * random.next())
                addEvent [agentId], null, (new Date(endTime)).toISOString(), "random-end " + o.description, {ends: startEvent}
                time += 2 * random.next() * 60 * 60 * 1000 / o.frequencyPerHour
        catch e
          console.log e
          warn "error creating random events for #{JSON.stringify o}: #{String(e)}"
    data.beforeRandom = {}

    behaviourApi = #{{{3
      addEvent: (o) ->
        if Array.isArray o.location
          addEvents o.agents, o.location, o.time, o.description, o
        else
          addEvent o.agents, o.location, o.time, o.description, o
      addAgent: (agent) ->
        agent.id = agent.id || uniqueId()
        warn "missing agent kind #{agent.id}" if !agent.kind
        warn "duplicate agent #{agent.id}" if data.agents[agent.id]
        data.agents[agent.id] = agent
        assignAgentColors(agent)
      randomEvents: randomEvents

    try (require "./#{config.configData}/behaviour.js").calendarAgents (data.calendar || []), behaviourApi, data
    catch e
      warn "Error in configuration in data/ " + e

    data.eventPos = 0 #{{{3
    data.agentNow = {}
    data.locationNow = {}
    data.eventList = Object.keys data.events
    data.eventList.sort()

    while data.eventPos < data.eventList.length && data.eventList[data.eventPos] < getDateTime()
      updateState(filterEvent(data.events[data.eventList[data.eventPos]]))
      data.eventPos += 1

    data.next = {}
    for event in data.eventList
      e = data.events[event]
      if e.location
        data.next[e.location] ?= []
        data.next[e.location].push event
      for agent in (e.agents || [])
        data.next[agent] ?= []
        data.next[agent].push event
    for id, events of data.next
      data.next[id] = events.reverse()

读取缓存数据

  try
    data = JSON.parse fs.readFileSync config.apiserver.cachefile
    if data.status && data.status.warnings
      for key, val of data.status.warnings
        status.warnings[key] = "data " + val
  catch e
    console.log "reading cached data:", e
    data =
      activities: {}
      calendarEvents: []
      groups: {}
      locations: {}
      status: {}
      teachers: {}
    warn "couldn't read cached data"
  process.nextTick enrichData

服务器

  app = express()
  app.use express.static "#{__dirname}/public"
  server = app.listen config.apiserver.port
  console.log "starting server on port: #{config.apiserver.port}"

REST服务器

  app.use (req, res, next) ->

没有缓存,如果服务器通过cdn

    res.header "Cache-Control", "public, max-age=0"

HEARTS

    res.header "Access-Control-Allow-Origin", "*"

无需告诉全世界我们正在运行的服务器软件, - 安全最佳实践

    res.removeHeader "X-Powered-By"
    next()

  app.all "/stop-server", (req,res)->
    res.end "ok, exiting"
    setImmediate -> process.exit 0

  nextEvent = (agentOrLocation) ->
    events = data.next[agentOrLocation]
    return null if !events
    events.pop() while events.length && (events[events.length - 1] < getDateTime())
    events[events.length - 1]

  defRest = (name, member) ->
    app.all "/next/:id", (req, res) ->
      event = nextEvent req.params.id
      res.json(if event then {event: event} else {})
      res.end()

    app.all "/#{name}/:id", (req, res) ->
      res.json data[member][req.params.id]
      res.end()
    app.all "/#{member}", (req, res) ->
      res.json Object.keys data[member]
      res.end()

  endpoints =
    teacher: "teachers"
    activity: "activities"
    group: "groups"
    location: "locations"
    event: "events"
    agent: "agents"

  updateStatus = (cb) ->
    fs.stat config.apiserver.cachefile, (err, stat) ->
      status.organismTime = getDateTime()
      status.lastDataUpdate = stat?.mtime
      status.eventDetails =
          count: data.eventList.length
          pos: data.eventPos
          first: data.eventList[0]
          next: data.eventList[data.eventPos + 1]
          last: data.eventList[data.eventList.length - 1]
      status.connections = clientCount
      cb? status
  updateStatus()

  app.all "/status", (req, res) ->
    updateStatus (result) ->
      res.json result
      res.end()

  app.all "/now/:kind/:id", (req, res) ->
    res.json (data[req.params.kind + "Now"] || {})[req.params.id] || {}
    res.end()

  app.all "/current-state", (req, res) ->
    res.json data.agentNow
    res.end()

  app.all "/client-config", (req, res) ->
    try
      file = __dirname + "/client-config/default.json"
      file = __dirname + "/client-config/user.json" if fs.existsSync(__dirname + "/client-config/user.json")
      res.json JSON.parse(fs.readFileSync(file, "utf8"))
    catch e
      res.json {}
    res.end()

  app.all "/arrivals", (req, res) ->
    arrivals (result) ->
      res.json result
      res.end()

  defRest name, member for name, member of endpoints

获取/更新请求时,将其写入data.json

例如上传时使用:curl -X POST -H“Content-Type:application / json”-d @ datafile.json http:// localhost:7890 / update

  app.all "/update", (req, res) ->
    handleUCCData req.body, -> res.end()
  app.use "/uccorg-update", (req, res, next) ->
    result = ""
    req.on "data", (data) ->
      result += data
    req.on "end", ->
      console.log "getting #{result.length} bytes"
      handleUCCData (JSON.parse result), -> res.end()

推送服务器

建立

  bayeux = new faye.NodeAdapter
    mount: '/faye'
    timeout: 45

  bayeux.on "subscribe", (clientId, channel) ->
    console.log channel, typeof channel

  clientCount = 0

  bayeux.on "handshake", (clientId, channel) ->
    ++clientCount

  bayeux.on "disconnect", (clientId, channel) ->
    --clientCount

  bayeux.attach server

事件和事件发射器

  filterEvent = (event) ->
    currentEvent = data.events[data.agentNow[event.agents[0]]?.event] || {}
    if eventType(event) == "random"

如果它们在可能发生的活动期间发生,则仅发出随机事件

      return if not (eventType(currentEvent) in event.during)

记住上一个事件,为了能够恢复它, - 但前提是它不是随机事件

      if eventType(currentEvent) != "random"
        data.beforeRandom[event.agents[0]] = currentEvent.id

    if eventType(event) == "random-end"
      if (data.agentNow[event.agents[0]]?.event) != event.ends
        data.beforeRandom[event.agents[0]] = undefined
        return
      prevEvent = data.events[data.beforeRandom[event.agents[0]] ]
      if prevEvent and !(prevEvent.description == "roaming" && prevEvent.autoLeave)
        event.description = prevEvent.description
        event.location = prevEvent.location
        event.clonedId = prevEvent.id
      else
        event.description = "roaming"
        event.autoLeave = leaveTime(event.id)

      data.beforeRandom[event.agents[0]] = undefined

    if event.description == "away" and data.agentNow[event.agents[0]].activity != "roaming"

如果做随机的东西,TODO也会消失

      return
    return event

  autoLeave = []
  nextAutoLeave = "0"
  addAutoLeave = (event) ->
    autoLeave.push event
    nextAutoLeave = event.autoLeave if event.autoLeave < nextAutoLeave
  handleAutoLeave = (now) ->
    events = []
    if nextAutoLeave <= now
      list = autoLeave
      autoLeave = []
      nextAutoLeave = "9999"
      for event in list
        if event.autoLeave <= now
          for agent in event.agents
            if data.agentNow[agent].event == event.id
              id = event.autoLeave + '_' + uniqueHash(agent + event.id) + '_away'
              data.events[id] =
                id: id
                description: "away"
                time: event.autoLeave
                agents: [agent]
              events.push id
        else
          addAutoLeave event
    events

  leaveTime = (t) ->
    prng = prand(hash(t))
    (new Date(new Date(t.slice(0,19)+'Z') - (- (generalSettings.minRoam + (generalSettings.maxRoam - generalSettings.minRoam) * prng.next())|0) * 60 * 1000)).toISOString().slice(0,19)

  emitEvent = (event) ->
    event = filterEvent event
    return if not event
    addAutoLeave(event) if event.autoLeave
    data.events[event.id] = event if !data.events[event.id]
    console.log getDateTime(), event.id, event.description, event.location
    updateState event
    bayeux.getClient().publish "/events", event

  eventEmitter = ->
    now = getDateTime()
    events = handleAutoLeave now
    while data.eventPos < data.eventList.length and data.eventList[data.eventPos] <= now
      events.push data.eventList[data.eventPos]
      ++data.eventPos
    events.sort()
    for event in events
      emitEvent data.events[event]

  setInterval eventEmitter, 100

从rejseplanen训练到达数据

获取数据

  arrivalCache = []
  getArrivals = (d, cb) ->

    url = "http://xmlopen.rejseplanen.dk/bin/rest.exe/arrivalBoard" +
      "?id=8600683&date=#{d.getUTCDate()}.#{d.getUTCMonth() + 1}.#{String(d.getUTCFullYear()).slice(2)}&time=#{d.getUTCHours()}:#{d.getUTCMinutes()}"
    (require "request") url, (err, _, data) ->
      return cb(err) if err
      arrivalCache = []
      !data.replace /<Arrival name="(.*?)"[^>]*?type="(.*?)"[^>]*?time="(.*?)" date="(.*?)" [^>]*? origin="(.*)">/g, (_, name, type, time, date, origin) ->
        arrivalCache.push
          name: name
          type: type
          date: "20#{date.slice(6,8)}-#{date.slice(3,5)}-#{date.slice(0,2)}T#{time}:00"
          origin: origin
      cb null, arrivalCache

  arrivals = (cb) ->
    now = getDateTime()
    getArrivals (new Date(now)), (err, result) ->
      if err
        cb []
      else
        cb result

发出事件

  lastArrivalEmit = undefined

  doArrival = (arrival) ->
    agentId = arrival.name + " " + arrival.origin
    agent = data.agents[agentId]
    if !agent
      agent = data.agents[agentId] =
        id: agentId
        kind: "transport"
        name: arrival.name
        origin: arrival.origin
    location = data.locations[arrival.type]
    if !location
      location = data.locations[arrival.type] =
        id: arrival.type
        kind: "transport"
    emitEvent
      id: getDateTime() + agentId + " arrive"
      time: getDateTime()
      description: "transport arrival"
      agents: [agent.id]
      location: arrival.type
    sleep 2 + Math.random() * 60, ->
      emitEvent
        id: getDateTime() + agentId + " leave"
        time: getDateTime()
        description: "transport leaving"
        agents: [agent.id]

  arrivalEmitter = ->
    now = getDateTime().slice(0,-6) + "00"

    doEmit = (arrs) ->
      if now == lastArrivalEmit
        return setTimeout arrivalEmitter, 30000
      lastArrivalEmit = now
      if !arrs.length
        return setTimeout arrivalEmitter, 60*60*1000
      for arrival in arrs
        if arrival.date == now
          doArrival arrival
      setTimeout arrivalEmitter, 30000

    if !arrivalCache.length || now >= arrivalCache[arrivalCache.length - 1].date
      arrivals doEmit
    else
      doEmit arrivalCache

  arrivalEmitter() if not config.test

更新全局状态(代理/事件)

  updateState = (event) ->
    return if not event
    for agent in event.agents
      prevLocation = (data.agentNow[agent] || {}).location
      data.locationNow[prevLocation].agents = data.locationNow[prevLocation].agents.filter ( (a) -> a != agent) if prevLocation
      location = event.location
      if location
        data.locationNow[location] = data.locationNow[location] || {}
        data.locationNow[location].agents = data.locationNow[location].agents || []
        data.locationNow[location].agents.push agent
        data.locationNow[location].event = event.id
      data.agentNow[agent] = {}
      data.agentNow[agent].location = location if location
      data.agentNow[agent].activity = event.description if event.description
      data.agentNow[agent].event = event.id

  calendarData enrichData #{{{2

测试

  if config.test
    testResult = ""
    testLog = (args...)->
      testResult += (JSON.stringify([args...]) + "\n").replace(/("id":"2015[^_]*)[^"]*/, '"id":"some-id')
    testDone = ->
      fs.writeFileSync config.test.outfile, testResult if config.test.outfile
      process.exit()



    testStart = config.test.startDate
    testEnd = config.test.endDate

在测试期间运行时间的因素

    testSpeed = config.test.xTime

休息测试

    restTest = ->
      restTest = -> undefined
      console.log "restTest", getDateTime()

      url = "http://localhost:#{config.apiserver.port}/"
      restTestRequest = (id) -> (done) ->
        request url + id, (err, req, data) ->
          testLog id, JSON.parse data
          done()
      async.series [
        restTestRequest "now/location/Brikserum C.125"
        restTestRequest "now/group/49"
        restTestRequest "now/teacher/23"
        restTestRequest "now/location/C.284"
        restTestRequest "group/49"
        restTestRequest "teacher/23"
        restTestRequest "location/C.208"
        restTestRequest "activity/99009"
      ]
      undefined

模拟getDateTime,

Date对应于测试数据集,以及运行速度非常快的时钟

    testTime = + (new Date testStart)
    getDateTime = -> testTime
    setTimeout ( ->
      startTime = Date.now()
      getDateTime = -> (new Date(testTime + (Date.now() - startTime) * testSpeed)).toISOString() 
    ), 3000

运行测试 - 当前测试客户端只是将“/ events”作为“/ test”发回

    bayeux.getClient().subscribe "/events", (message) ->
      testLog "event", message
    setInterval (->
      if config.test.restTestTime && getDateTime() >= config.test.restTestTime
        restTest()
      if getDateTime() >= testEnd
        testLog "testDone"
        testDone()
    ), 100000 / testSpeed

sendUpdate数据, - >未定义


自动生成README.md,编辑uccorg-backend.coffee进行更新

本文使用googletrans自动翻译,仅供参考, 原文来自github.com

en_README.md

UCC Organism Backend

ci

Backend for the UCC-organism

Info

The server is run with coffee uccorg-backend.coffee configfile.json, where configfile.json contains the actual configuration of the server.
Depending on the configuration, this runs as:

  • production data preparation server(windows), which is responsible for getting the data from ucc/webuntis/calendar/..., anonymising them, and sending them onwards to the api server
  • production api server(debian on macmini), which gets the anonymised data from the data preparation server, makes them available via an api, and emits events using the Bayeux protocol
  • development server for backend, which uses real data dumps instead of talking with external services
  • automated test, which runs automatically using travis, uses sample data dumps, and mocks the time to run very fast.
  • development server for frontend, - which runs of sample data dump and mocks the time, - to be able to get events without having to wait for real-world activities

Configuration

All configuration options are listed in config.json.sample. Also see test.json for an actual configuration, the content of this configuration wille also be a good choice for a frontend development server, - just remove "outfile", and reduce the time speed factor "xTime" - which tells how much faster the mocked clock should run.

Client config pushed on deman to the Odroids / Screens is in client-config/default.js

API

The api delivers JSON objects, and is available through http, with JSONP and CORS enabled. The endpoints are:

  • /(agent|location|event)/$ID returns info about the particular entity
  • /now/(agent|location)/$ID returns an object with status for the moment, NOTICE: this varies over time
  • /current-state returns active agents+activities
  • /client-config returns client config

Old api:
- /(agents|locations|events) returns list of ids. NOTICE: This is meant for development/exploration, and not expected to be used in production, as some events/agents will be created on the fly and may not be in the list yet
- /(teacher|group|activity)/$ID returns info about the particular entity
- /(teachers|groups|activities) returns list of ids

Events are pushed on /events as they happens through faye (http://faye.jcoglan.com/), ie. (new Faye.Client('http://localhost:8080/')).subscribe('/events', function(msg) { ... })

uniform agent scheduling notes

Data schema:
- agents: (teachers, or students member of groups)
- id
- kind: teacher | student | researcher | janitor | bus | train | ...
- ?gender: 0/1 (for teacher/student)
- ?groups of ids (for students)
- ?programme:
- ?age (for student)
- ?end (for student)
- ?name (for train/bus)
- ?origin (for train/bus)
- events - from activities, train arrivals, etc.
- time
- TODO kind: activity-start, activity-end, bus-arrival, train-arrival,
- agents
- ?description: activity-subject, etc.
- ?location
- TODO ?activity - link to ucc-activity for debugging
- locations
- id
- name
- ?capacity

Production server setup

For the production system we choose to use a virtual linux server on the existing mac.

Requirements for the server for the api-server:

  • boots automatically on power failure etc.
  • runs linux (other operating systems should also work, but I will not give any support on it)
  • when running a http service on port 8080, it will be available as http://10.251.26.11:8080 on the internal network
  • some way of rebooting it, and accessing it, remotely

The immediate options are either

  • to set up a dedicated server on the UCC network
  • run a virtual server on the mac that will drive one of displays

As there already is a virtual server configured on the mac, and under assumption that it is being kept running in production, we choose the latter.

Server setup on linux, as user uccorganism with home /home/uccorganism. Assumes git/nc is installed via package manager, and node/npm/coffee is installed in /usr/local/bin. Installation of the server is done with:

cd
git clone https://github.com/UCC-Organism/uccorg-backend.git
cd uccorg-backend
npm install
(crontab -l; echo @reboot /home/uccorganism/uccorg-backend/apiserver-start.sh) | sort | uniq | crontab -

Then the api-server should then run after a reboot

Assumptions about external data sources

Data sources

  • webuntis access via an API-key
  • /api/[locations|subjects|lessons|groups|teachers|departments] returns json array of ids
  • /api/locations/$ID returns json object with: untis_id, capacity, longname
  • /api/subjects/$ID returns json object with: untis_id, name, longname, alias
  • /api/lessons/$ID returns json object with: untis_id, start, end, subjects, teachers, groups, locations, course
  • /api/groups/$ID returns json object with: untis_id, name, alias, schoolyear, longname, department
    • The alias must match the "Holdnavn" in the mssql database
  • /api/teachers/$ID returns json object with: untis_id, name, forename, longname, departments
    • The name must match "Initialer" in the mssql-database
  • /api/departments/$ID returns json object with: untis_id, name, longname
  • ucc mssql database - accessible via stored procedures on the server:
  • GetStuderendeHoldCampusNord: Holdnavn (=groups.alias), Studienummer
  • GetAnsatteHoldCampusNord: Holdnavn, Initialer
  • GetAnsatteCampusNord: Initialer, Afdeling, Køn
    • There must be an entry for each teacher from webuntis, with Initialer the same as teacher.name
  • GetStuderendeCampusNord: Studienummer, Afdeling, Køn(0/1 -> agent.gender), Fødselsdag(DDMMYY -> agent.gender)
  • GetHoldCampusNord: Afdeling, Holdnavn, Beskrivelse, StartDato, SlutDato
    • There should be a hold for each Holdnavn for ansatte/studerende, and also one matching the every groups.alias from webuntis
  • google-calendar returns ical data parseable by rrule, with SUMMARY as the title of the event
  • rejseplanen-api http://xmlopen.rejseplanen.dk/bin/rest.exe/arrivalBoard?id=8600683&date=... returns schedule with arrivals in the form <Arrival name="..." type="..." date="..." origin="...">.

Status

Release Log

January-April 2015

  • Overordnet
  • √homogen repræsentation af alle agent-typer, så eksempelvis forskere, undervisere, pedeller, køkkenersonale etc. repæsenteres på samme måde som studerende: tilknyttes grupper, bevæger sig mellem lokaler etc.
  • √tilfældig opførsel af agenter, såsom pauser mellem undervisning, toiletbesøg, frokost etc.
  • √globale tilstande såsom: dagscyklus, udbetaling af su og lignende
  • √mulighed for at konfigurere tilfældig opførsel og events
  • √løbende tilpasninger af backend efter ønsker fra A&K og frontendudviklingen frem til idiftsættelsen
  • √håndtering af at systemet kører videre, selv hvis de eksterne datakilder fejler
  • √afklaring af driftskonfiguration, - skal vi sætte en separat linux-server op, eller køre det parallelt på mac'en der også driver skærmen
  • √not-needed (eventult konfiguration og opsætning af linux-server)
  • √proaktiv løbende kommunikation med frontendudviklingen, for at sikre at backend matcher ønsker og forventninger i forhold til frontend
  • √dokumentation af forventninger og krav til de eksterne datakilder

  • week 16

  • intensity level 0-1+random for globale events
  • forskudt tid (håndterer skævt ur på mac)
  • week 15
  • case insensitive+trim calendar events
  • documentation of used data sources in README
  • to tilfældige farver fra colorrange+fagretning per agent.
  • week 14
  • bugfix: neverending roaming
  • decide-on/document production setup
  • configurable shortening of classes to create breaks
  • configurable configuration+calendar + static data for repeatable unit test, when system configuration changes
  • configure events for administrators, janitors, kitchenstaff, and cafe/canteen
  • fix pseudorandom - bug due to imul
  • (frontend running on linux)
  • week 13
  • random behaviour
    • only random events on agents present
    • random event emission - background and restore other events
    • generate random events
    • configurable random behaviour - lunch, toilet-break, illness-leave, pauser mellem undervisning etc.
    • random-hash-evenly-distributed
    • sample random configuration passed into the system
    • prefix for events from schedule + isScheduled
    • refactor next to be a function
  • more url-friendly event-ids (with "_" instead of " ")
  • event-id in /now
  • have events also bring along a likely end time
  • implement /next api endpoint
  • week 12
  • add random roaming/away state for agents - needed for random events for agents at campus not doing anything in particular
    • configurable in data/general-settings.json
  • server fejlbesked hvis fejl i json
  • week 10
  • global state - day cycle etc. via agent - ie. /agent/time-of-day day cycle - grants, su, etc. configurable
  • apiserver-script in version control
  • support for stopping/rebooting the API-server remotely
  • calendar data retrieval in API-server
  • internal: preserve order of event-ids using hash function, to avoid test error due to changing order of events at same time.
  • week 9
  • change structure of configuration files in data/ to make them easier to edit: data/behavior.js split up into locations.json, agents.json, calendar.js, og behaviour.js
  • begun moving calendar-retrieval from data-processing to API-server
  • week 8
  • repeat with old data, if we haven't gotten any updates from the data server recently
  • week 7
  • handle several locations, by distributing agents into locations
  • creation of events and agents from calendar
  • configuration of calendar behaviour in data/behaviour.js
  • week 6
  • include warnings in status, and propagate warnings from windows server to api-server
  • bus/train events as uniform events instead of separate arrivals
  • contact UCC about SSMLDATA-server is down
  • some refactoring / dead code elimination
  • week 5
  • webservice to get current state of agents and locations, replaces old /now/
  • events emitted are from the new uniform api
  • new test data (but rest-test disabled)
  • week 4
  • draft new api: /events, /agents, /locations
  • agent+event uniform static data
  • temporary proxy for frontend development

In between development 2014

  • dummy-hold when missing data
  • workaround for data where several groups has the same untis_did
  • create dummy data with "error:missing" when missing teacher, location or subject
  • student age
  • ignore bad ssl-certificates for webuntis - as they were/(are?) buggy

Milestone 3 - running until Jan 20. 2014

  • configure mac-mini autostart api-server
  • dashboard
  • get data from remote-calendar (train schedule, etc.)
  • fix timezone bug (test daylight saving handling)
  • do not tunnel data anymore, but send it directly to the macmini via port 8080 now that the firewall is opened.
  • update config on windows server, to send current days, and not one month in the future for test.
  • preparation-server: support dump to file for development purposes
  • dashboard: show events live as they happen
  • dashboard skeleton
  • added api for getting ids of all teachers/groups/locations/activities

Milestone 2 - running until Dec. 29 2013

  • the windows server configured to extract the data each night at 1'o'clock, and send them to the mac mini.
  • added api for getting current/next/prev activity given a location, teacher or group
  • update REST-api
  • moving configuration into config-file
  • generate datafile for apiserver from ucc/webuntis-data
  • anonymising students
  • temporarily forwarding data through ssl.solsort.com, as port 8080 from ssmldata to macmini doesn't seem to be open.
  • send data from ssmldata-server to macmini

Milestone 1 - running until Dec. 22 2013

  • get data from sqlserver
  • getting data from webuntis
  • got macmini onto UCCs serverpark
  • live events via faye
  • get data via api
  • got access to provisioned ssmldata-server
  • dummy data-set for automatic test
  • automatic test on macmini with fast time for development
  • deprecated solsort.com running webuntis-based webservice for development

Data Issues

  • some teachers on webuntis missing from mssql (thus missing gender non-critical)
  • mapning mellem de enkelte kurser og hold mangler, har kun information på årgangsniveau, og hvilke årgange der følger hvert kursus
  • Info følgende grupper mangler via mssql: fss12b, fss11A, fss11B, fsf10a, fss10, fss10b, fss12a, norF14.1, norF14.2, norF14.3, nore12.1, samt "SPL M5 - F13A og F13B"
  • activity is not necessarily unique for group/location at a particular time, this slightly messes up current/next activity api, which just returns a singlura next/previous
  • navngivning af lokaler er måske ikke konsistent

Main

See sample file in config.json-sample, and test.json.

config = {}
if require.main == module then do ->
  try
    configfile = process.argv[2]
    configfile = "config" if !configfile
    configfile += ".json" if configfile.slice(-5) != ".json"
    config = JSON.parse (require "fs").readFileSync configfile, "utf8"
  catch e
    console.log "reading config #{configfile}:", e
    process.exit 1

  config.icalCacheFile ?= "cached-calendar.ical"
  config.configData ?= "data"

  process.nextTick ->
    if config.prepare then dataPreparationServer() else apiServer()

Common stuff

About

exports.about =
  title: "UCC Organism Backend"
  author: "Rasmus Erik Voel Jensen"
  description: "Backend for the UCC-organism"
  owner: "UCC-Organism"
  name: "uccorg-backend"
  dependencies:
    solapp: "*"
  package:
    scripts:
      test: "solapp build; rm -f test.out ; ./node_modules/coffee-script/bin/coffee uccorg-backend.coffee test ; diff test.out test.expected"
    dependencies:
      async: "0.2.9"
      "coffee-script": "1.6.3"
      express: "3.4.6"
      faye: "1.0.1"
      mssql: "0.4.1"
      request: "2.30.0"
      rrule: "2.0.0"
      "js-beautify": "1.5.4"
      "jshint": "2.6.0"

Dependencies

assert = require "assert"
fs = require "fs"
express = require "express"
http = require "http"
faye = require "faye"
async = require "async"
mssql = require "mssql"
request = require "request"

Utility functions

Math.imul

Math.imul ?= (a,b)->
  _ah = (a >>> 16) & 0xffff
  _al = a & 0xffff
  _bh = (b >>> 16) & 0xffff
  _bl = b & 0xffff
  (_al * _bl) + (((_ah * _bl + _al * _bh) << 16) >>> 0)|0

evenType

eventType = (eventObject) -> (eventObject.description || "undefined").split(" ")[0]

djb2-hash

hash = (str) ->
  result = 5381
  for i in [1..str.length-1]
    result = 33 * result + str.charCodeAt(i) &0x7fffffff
  result

uniqueHash

uniqueHashes = {}
uniqueHash = (s) ->
  h = hash(s)
  s2 = uniqueHashes[h]
  while s2 && s2 != s
    h += 1
    s2 = uniqueHashes[h]
  uniqueHashes[h] = s
  h

pseudoRandom utility functions

prand = (i) ->
  i ?= 0
  next = -> 
    i = Math.imul(1103515245, i) + 12345 & 0x7fffffff
  return {
    next: -> next(); (i & 0x3fffffff) / 0x40000000
    nextN: (n) -> next(); n * (i & 0x3fffffff) / 0x40000000 | 0
  }

pseudorandom

seed = prand(0)
pseudoRandom = -> seed.next()

uniqueId

uniqueId = do ->
  prevId = 0
  return -> prevId += 1

getDateTime

Get the current time as yyyy-mm-ddThh:mm:ss (local timezone, - or mocked value if running test/dev)

timeoffset = 0
timecorrection = 0
getDateTime = -> (new Date(Date.now() + timecorrection - (new Date).getTimezoneOffset() * 60 * 1000 + timeoffset)).toISOString().slice(0,-1)

Correct time if very wrong

Get time from http. Notice this is very imprecise, and the server setup should find the correct time via NTP. Unfortunately we seem to be behind a restrictive firewall, so if we can see that the local clock is drifting way wrong, we try to work around it.

if require.main == module then setImmediate ->
  (http.get "http://www.google.com", (o) ->
    requestTime = Date.now()
    remoteTime = new Date(o?.headers?.date)
    return warn "error getting date from www.google.com for clock sanity check" if String(remoteTime) == "Invalid Date"

This is very imprecise, requestTime occured at remoteTime + network latency
and we do not know the network latency (and http-request include handshake,
so it is also difficult to estimate).

Anyhow remoteTime has precision in seconds, and network latency also
shouldn't be many seconds, so the offset should be precise within
a couple of seconds, - and we only apply it if we can observe a drift
of more than ½ minute :(

    timeOffset = (+remoteTime) - requestTime

    if Math.abs(timeOffset) > 30*1000
      warn "servertime is more the 30s off (#{timeOffset / 1000}), compared to google webservers + network latency, - local clock is probably very drifting, and thus we apply imprecise corrections"
      timecorrection = timeOffset
  ).on("error", (-> warn "error connecting to www.google.com for clock sanity check"))

sleep

more comfortable syntax for set timeout: sleep #seconds, -> ...

sleep = (t, fn) -> setTimeout fn, t*1000

binarySearchFn

binSearchFn = (arr, fn) ->
  start = 0
  end = arr.length
  while start < end
    mid = start + end >> 1
    if fn(arr[mid]) < 0
      start = mid + 1
    else
      end = mid
  return start

if typeof isTesting == "boolean" && isTesting then do ->
  arr = [0,1,2,3,4,5]
  assert.equal 2, binSearchFn arr, (a) -> a - 2
  assert.equal 3, binSearchFn arr, (a) -> a - 2.1

sendUpdate

sendUpdate = (data, callback) ->
  datastr = JSON.stringify data

escape unicode as ascii

  datastr = datastr.replace /[^\x00-\x7f]/g, (c) ->
    "\\u" + (0x10000 + c.charCodeAt(0)).toString(16).slice(1)
  opts =
    hostname: config.prepare.dest.host
    path: config.prepare.dest.path || "/uccorg-update"
    port: config.prepare.dest.port || undefined
    method: "post"
    headers:
      "Content-Type": "application/json"
      "Content-Length": datastr.length
  if !(config.prepare.dest.protocol in ["http", "https"])
    throw "config error: prepare.dest.protocol neither 'http' nor 'https'" 
  req = (require config.prepare.dest.protocol).request opts, callback
  console.log "sending data: #{datastr.length} bytes"
  req.write datastr
  req.end()

status

status =
  bootTime: getDateTime()
  warnings: {}
warn = (msg) ->
  status.warnings[msg] = getDateTime()

generalSettings =
  try (require "./#{config.configData}/general-settings.json")
  catch e
    warn "Error in configuration in #{config.configData}/ " + e
    {}

generalSettings.minRoam ?= 15
generalSettings.maxRoam ?= 30

calendar

data preparation - processing/extract running on the SSMLDATA-server

dataPreparationServer = ->

SQL Server data source

  getSqlServerData = (done) ->
    if config.prepare.mssqlDump
      try
        result = JSON.parse fs.readFileSync config.prepare.mssqlDump
        return done? result
      catch e
        console.log "Loading mssql dump:",  e

    entries = ["Hold", "Studerende", "Ansatte", "AnsatteHold", "StuderendeHold"]
    result = {}
    return done?(result) if not config.prepare.mssql

    handleNext = ->
      if entries.length == 0
        if config.prepare.mssqlDump
          fs.writeFileSync config.prepare.mssqlDump, (JSON.stringify result, null, 2)
        return done?(result)
      current = entries.pop()
      console.log "mssql", current
      req = con.request()
      req.execute "Get#{current}CampusNord", (err, reqset) ->
        throw err if err
        result[current] = reqset
        handleNext()

    con = new mssql.Connection config.prepare.mssql, (err) ->
      throw err if err
      handleNext()

Webuntis data source

We do not yet know if we should use the webuntis api, or get a single data dump from ucc
If needed extract code from old-backend-code.js

  getWebUntisData = (callback) ->
    if config.prepare.webuntisDump
      try
        result = JSON.parse fs.readFileSync config.prepare.webuntisDump
        return callback?(result)
      catch e
        warn "error loading webunits dump"
        console.log "Loading webuntis dump:", e

    do ->
      apikey = config.prepare.webuntis
      untisCall = 0

webuntis - function for calling the webuntis api

      webuntis = (name, cb) ->
        if ((++untisCall) % 100) == 0
          console.log "webuntis api call ##{untisCall}: #{name}"
        url = "https://api.webuntis.dk/api/" + name + "?api_key=" + apikey
        request {
            url: url
            rejectUnauthorized: false
          }, (err, result, content) ->
            warn "webuntis request error #{name}" if err
            return cb err if err
            cb null, JSON.parse content

extract data, download data needed from webuntis

      extractData = (dataDone) ->
        result =
          locations: {}
          subjects: {}
          lessons: {}
          groups: {}
          teachers: {}
          departments: {}

        async.eachSeries (Object.keys result), ((datatype, cb) ->
          console.log "getting #{datatype} from webuntis"
          webuntis datatype, (err, data) ->
            cb err if err
            async.eachSeries data, ((obj, cb) ->
              id = obj.untis_id
              webuntis "#{datatype}/#{id}", (err, data) ->
                result[datatype][id] = data
                cb err
            ), (err) -> cb err
        ), (err) ->
          dataDone err, result

      extractData (err, data) ->
        warn "extractData error" if err
        throw err if err
        if config.prepare.webuntisDump
          fs.writeFileSync config.prepare.webuntisDump, (JSON.stringify data, null, 2)
        callback?(data)

Transform data for the event/api-server

  processData = (webuntis, sqlserver, callback) ->
    startTime = config.prepare.startDate || 0
    if typeof startTime == "number"
      startTime = (new Date(+(new Date()) + startTime * 24*60*60*1000)).toISOString()
    else
      startTime = (new Date(startTime)).toISOString()
    endTime = (new Date(+(new Date(startTime)) + (config.prepare.timespan || 1) * 24*60*60*1000)).toISOString()
    console.log "Extracting data from #{startTime} to #{endTime}"

The file in the repository contains sample data for test.

For each kind of data there is a mapping from id to individual object

Output description

  • activities: id, start/end, teachers, locations, subject, groups
  • groups: id, group-name, programme, students(id,gender)
  • teachers: id, gender, programme
  • locations: id, name, longname, capacity
  • calendarEvents: start, end, title, description

Initialisation

    result =
      locations: {}
      activities: {}
      groups: {}
      teachers: {}

Locations

    for _, location of webuntis.locations
      result.locations[location.name] =
        id: location.name
        name: location.longname
        capacity: location.capacity

addTeacher

    teachers = {}
    for obj in sqlserver.Ansatte[0]
      teachers[obj.Initialer] = obj

    addTeacher = (obj) ->
      id = obj.untis_id
      name = obj.name
      warn "missing teacher in mssql" if !teachers[name]
      result.teachers[id] =
        id: id
        gender: teachers[name]?["Køn"]
        programme: teachers[name]?["Afdeling"]
        programmeDesc: do ->
          id = obj.departments[0]
          dept = webuntis.departments[id]
          "#{dept?.name} - #{dept?.longname}"

addGroup (and students)

    students = {}
    studentId = 0
    studentIds = {}
    getStudentId = (studienummer) -> studentIds[studienummer]

    for obj in sqlserver.Studerende[0]
      studentIds[obj.Studienummer] = ++studentId
      studentObject =
        id: getStudentId obj.Studienummer
        gender: obj["Køn"]

calculate age from birthday

      today = new Date()
      birthday = obj["Fødselsdag"]
      if birthday

        birthyear = parseInt(birthday.slice(4,6), 10)

fix two-digit year problem,
ie. "12" could be both 1912 and 2012
assume 21st century if it is before today.

        birthyear += 100 if birthyear < (today.getYear() - 100)

        birthmonth = parseInt(birthday.slice(2,4), 10)
        birthdate = parseInt(birthday.slice(0,2), 10)

        age = today.getYear() - birthyear
        age -= 1 if new Date(today.getYear(), birthmonth - 1, birthdate) > today

        studentObject.age = age

      studentObject.end = obj.Forventet_slutdato if obj.Forventet_slutdato

      students[getStudentId obj.Studienummer] = studentObject

    groups = {}
    for obj in sqlserver.Hold[0]
      groups[obj.Holdnavn] =
        name: obj.Holdnavn
        department: obj.Afdeling
        start: obj.StartDato
        end: obj.SlutDato
        students: []
    for obj in sqlserver.StuderendeHold[0]
      if !groups[obj.Holdnavn]
        console.log 'Data error: Hold missing for StuderendeHold', obj
        warn 'Data error: Hold missing for StuderendeHold ' + obj.Holdnavn

Dummy hold

        groups[obj.Holdnavn] =
          name: obj.Holdnavn
          department: '219405'
          start: '01-01-2014'
          end: '01-01-2060'
          students: []
      groups[obj.Holdnavn].students.push students[getStudentId obj.Studienummer]

    addGroup = (obj) ->
      return obj.untis_id if result.groups[obj.untis_id]

Buggy data: sometimes the same group has several untis_id, which is why we make a deep copy with json.parse/stringify

      grp = result.groups[obj.untis_id] = JSON.parse JSON.stringify groups[obj.alias] || {}
      grp.id = obj.untis_id
      grp.group = obj.name
      dept = webuntis.departments[obj.department]
      warn "Department missing: #{obj.department}" if !dept and obj.department
      grp.programme = "#{dept?.name} - #{dept?.longname}"
      grp.id

Add all groups groups and teachers

to make sure they are available if needed needed by calendar events

    do ->
      for _, teacher of webuntis.teachers
        addTeacher teacher

      for _, group of webuntis.groups
        addGroup group

Handle Activities

    for _, activity of webuntis.lessons
      if startTime < activity.end && activity.start < endTime && activity.end
        result.activities[activity.untis_id] =
          id: activity.untis_id
          start: activity.start
          end: activity.end
          teachers: activity.teachers.map (untis_id) ->
            addTeacher webuntis.teachers[untis_id] || {untis_id: untis_id, name: "error:missing", departments: ["error:missing"] }
            untis_id
          locations: activity.locations.map (loc) -> webuntis.locations[loc]?.name || "error:missing"
          subject: activity.subjects.map((subj) -> webuntis.subjects[subj]?.longname || "error:missing").join(" ")
          groups: activity.groups.map (untis_id) ->
            addGroup webuntis.groups[untis_id] || {untis_id: untis_id}

done

    callback result

execute

  getWebUntisData (data1) ->
    getSqlServerData (data2) ->
      processData data1, data2, (result) ->
        result.status = status
        if config.prepare.dest.dump
          fs.writeFile config.prepare.dest.dump, JSON.stringify(result, null, 2)
        sendUpdate result, (err, data) ->
          if err
            console.log 'sendUpdate error:', err
            warn 'sendUpdate error'
          console.log "submitted to api-server"
          process.exit 0

event/api-server

apiServer = ->
  data = undefined
  activitiesBy =
    group: {}
    location: {}
    teacher: {}
  eventsByAgent = {}

calendarData

  calendarData = (done) ->
    icaldata = []

    request config.icalUrl || "http://no-ical-url-in-config/", (err, result, ical) ->
      if err
        warn 'Error getting calendar data ' + config.icalUrl
        ical = fs.readFileSync config.icalCacheFile, "utf8"
      else
        fs.writeFile config.icalCacheFile, ical

      return done() if !ical

      icaldata = []
      !ical.replace /BEGIN:VEVENT([\s\S]*?)END:VEVENT/g, (_,e) ->
        props = e.split(/\r\n/).filter((x) -> x != "")
        event = {}
        for prop in props
          pos = prop.indexOf ":"
          pos = Math.min(pos, prop.indexOf ";") if prop.indexOf(";") != -1
          event[prop.slice(0,pos)] = prop.slice(pos+1)
        icaldata.push event

      calId = 0
      result = []
      startTime = getDateTime()
      endTime = +(new Date(getDateTime())) + 7 * 24 * 60 * 60 * 1000
      result.calendarEvents = []

      handleEvent = (dtstart, event) ->
        activity =
          id: "cal#{++calId}"
          start: dtstart.toISOString()
          end: new Date(+dtstart + (+iCalDate(event.DTEND) - +iCalDate(event.DTSTART))).toISOString()
          type: String(event.SUMMARY).trim().toLowerCase()
        result.push activity

      iCalDate = (t) ->
        d = t.replace /.*:/, ""

WARNING: here we assume that we are in Europe/Copenhagen-timezone

        d = new Date(+d.slice(0,4), +d.slice(4,6) - 1, + d.slice(6,8), +d.slice(9,11), +d.slice(11,13), +d.slice(13,15), 0)
        d = new Date(+d - d.getTimezoneOffset() * 60 * 1000)
        if (t.slice(0, 23) == "TZID=Europe/Copenhagen:") || (t.slice(0,11) == "VALUE=DATE:")
          d
        else if t.slice(-1) == "Z"
          d = new Date(+d - d.getTimezoneOffset() * 60 * 1000)
        else
          warn "timezone bug in calendar data " + t + " " + d
          console.log "timezone bug in calendar data", t, d
        d

      if icaldata then for event in icaldata
        if event.RRULE
          RRule = (require "rrule").RRule
          opts = RRule.parseString event.RRULE
          opts.dtstart = iCalDate event.DTSTART
          rule = new RRule(opts)
          occurences = rule.between(new Date(startTime), new Date(endTime), true)
          for occurence in occurences
            handleEvent occurence, event
        else if startTime <= iCalDate(event.DTSTART).toISOString() < endTime
          handleEvent iCalDate(event.DTSTART), event

      data.calendar = result if data?
      done result

calculate time offset once per hour, rewind clock a week, if more than eight days since last data update

  updateTimeOffset = ->
    lastUpdate = new Date(status.lastDataUpdate)
    if 8 < (new Date() - lastUpdate + timeoffset) / 1000 / 60 / 60 / 24
      timeoffset -= 7 * 24 * 60 * 60 * 1000
      enrichData()
  setInterval updateTimeOffset, 1000 * 60

Handle data

Pushed to the server from UCC daily.

  handleUCCData = (input, done) ->
    console.log "handling data update from ucc-server"
    fs.writeFile config.apiserver.cachefile, JSON.stringify(input), ->
      data = input
      if input.status and input.status.warnings
        for key, val of input.status.warnings
          status.warnings[key] = "data " + val
      calendarData enrichData
      console.log "data replaced with new data from ucc-server"
      done()

  enrichData = -> #{{{3

    activitiesBy =
      group: {}
      location: {}
      teacher: {}

Tables with activities ordered by group/location/teacher

    for _, activity of data.activities
      for kind, collection of activitiesBy
        for elem in activity["#{kind}s"]
          collection[elem] ?= []
          collection[elem].push activity
    for _, collection of activitiesBy
      for _, arr of collection
        arr.sort (a, b) -> a.end.localeCompare b.end

add agent+events

agent/event data structure

    assignAgentColors = (agent) ->
      seed = prand(hash(agent.id))

      randomComponent = (a, b) ->
         a = parseInt a, 16
         b = parseInt b, 16
         n = 0x100 + a + seed.next() * (b - a) | 0
         n.toString(16).slice(1)

      randomColor = (c1, c2) ->
        r = randomComponent(c1.slice(1,3),c2.slice(1,3))
        g = randomComponent(c1.slice(3,5),c2.slice(3,5))
        b = randomComponent(c1.slice(5,7),c2.slice(5,7))
        "#" + r + g + b

      colors = generalSettings.agentColors?[agent.programme]
      colors = generalSettings.agentColors?.default if !colors
      return if !(colors?.color1min and colors.color2min and colors.color1max and colors.color2max)

      agent.color1 = randomColor(colors.color1min, colors.color1max)
      agent.color2 = randomColor(colors.color2min, colors.color2max)

    data.agents = {} #{{{3
    for _, teacher of data.teachers
      id = "teacher" + teacher.id
      data.agents[id] = agent = {}
      agent.kind = "teacher"
      agent.gender = teacher.gender
      agent.programme = teacher.programmeDesc
      agent.id = id
      assignAgentColors(agent)

    data.agents.JamesBond =
      kind: "yes"
      gender: 1
      license: "kill -9"
      id: "007"
      description: "undercover testagent"

    for groupId, group of data.groups
      continue if not group.students
      for student in group.students
        id = "student" + student.id
        data.agents[id] = agent = data.agents[id] || {}
        if agent.programme && group.programme != agent.programme
          console.log "warning: student in several programmes, ignoring", id, group.programme, agent.programme
          warn "warning: student in several programmes, ignoring " + id  + " " + group.programme + " " + agent.programme
        agent.kind = "student"
        agent.programme = group.programme
        agent.groups ?= []
        agent.groups.push groupId
        agent.age = student.age
        agent.gender = student.gender
        agent.end = student.end
        agent.id = id
        assignAgentColors(agent)

    data.events = {} # {{{3

    addEvent = (agents, location, time, description, misc) ->
      if !agents || !agents.length
        return

id = time + '' + hash("" + agents + location + description) + ''+ uniqueId()

      id = time + '_' + uniqueHash("" + agents + location + description) + '_'+ ((description || "").split " ")[0]
      data.events[id] =
        id: id
        location: location || undefined
        description: description
        time: time
        agents: agents
      if misc
        for key, val of misc
          data.events[id][key] ?= val
      id

    addEvents = (agents, locations, time, description, misc) ->
      len = locations.length
      if len > 1

distribute agents into locations for event

        for i in [0..len-1] by 1
          addEvent (agents[j] for j in [i..agents.length-1] by len),
            locations[i], time, description, misc
      else
        addEvent agents, locations[0], time, description, misc


    seed = prand(0)
    for _, activity of data.activities
      agents = []
      for teacherId in activity.teachers
        agents.push "teacher" + teacherId
      for groupId in activity.groups
        for student in data.groups[groupId].students || []
          agents.push "student" + student.id

      addEvents agents, activity.locations, activity.start, "scheduled " + activity.subject, { likelyEndTime: activity.end}
      minBreak = config.minBreak || 1 / 59
      maxBreak = config.maxBreak || 1 / 59
      breakRandom = prand(hash("" + activity.start + activity.locations + activity.agents))
      breakTime = 1000*60*(minBreak+breakRandom.next() * (maxBreak - minBreak))
      endTime = (new Date(new Date(activity.end.slice(0,19)+'Z') - breakTime)).toISOString().slice(0,19)
      addEvent agents, undefined, endTime, "roaming", {autoLeave: leaveTime(endTime + agents)}

for agent in agents
addEvent [agent], undefined, (new Date(new Date(activity.end.slice(0,19)+'Z') - (- (generalSettings.minRoam + (generalSettings.maxRoam - generalSettings.minRoam) * pseudoRandom())|0) * 60 * 1000)).toISOString().slice(0,19), "away"

    randomEvents = (start, end, events) -> #{{{3
      for o in events
        o.minDuration ?= 0.1
        o.maxDuration ?= 10
        try
          for agentId, agent of data.agents
            if agent.kind in o.agentTypes
              time = +(new Date(start))
              end = +(new Date(end))
              random = prand(hash(agentId + start + o.description))
              time += 2 * random.next() * 60 * 60 * 1000 / o.frequencyPerHour
              while time < end
                location = o.locations[random.nextN o.locations.length]
                startEvent = addEvent [agentId], location, (new Date(time)).toISOString(), "random " + o.description, {during: o.during}
                endTime = time + 1000*60* (o.minDuration  + (o.maxDuration - o.minDuration) * random.next())
                addEvent [agentId], null, (new Date(endTime)).toISOString(), "random-end " + o.description, {ends: startEvent}
                time += 2 * random.next() * 60 * 60 * 1000 / o.frequencyPerHour
        catch e
          console.log e
          warn "error creating random events for #{JSON.stringify o}: #{String(e)}"
    data.beforeRandom = {}

    behaviourApi = #{{{3
      addEvent: (o) ->
        if Array.isArray o.location
          addEvents o.agents, o.location, o.time, o.description, o
        else
          addEvent o.agents, o.location, o.time, o.description, o
      addAgent: (agent) ->
        agent.id = agent.id || uniqueId()
        warn "missing agent kind #{agent.id}" if !agent.kind
        warn "duplicate agent #{agent.id}" if data.agents[agent.id]
        data.agents[agent.id] = agent
        assignAgentColors(agent)
      randomEvents: randomEvents

    try (require "./#{config.configData}/behaviour.js").calendarAgents (data.calendar || []), behaviourApi, data
    catch e
      warn "Error in configuration in data/ " + e

    data.eventPos = 0 #{{{3
    data.agentNow = {}
    data.locationNow = {}
    data.eventList = Object.keys data.events
    data.eventList.sort()

    while data.eventPos < data.eventList.length && data.eventList[data.eventPos] < getDateTime()
      updateState(filterEvent(data.events[data.eventList[data.eventPos]]))
      data.eventPos += 1

    data.next = {}
    for event in data.eventList
      e = data.events[event]
      if e.location
        data.next[e.location] ?= []
        data.next[e.location].push event
      for agent in (e.agents || [])
        data.next[agent] ?= []
        data.next[agent].push event
    for id, events of data.next
      data.next[id] = events.reverse()

read cached data

  try
    data = JSON.parse fs.readFileSync config.apiserver.cachefile
    if data.status && data.status.warnings
      for key, val of data.status.warnings
        status.warnings[key] = "data " + val
  catch e
    console.log "reading cached data:", e
    data =
      activities: {}
      calendarEvents: []
      groups: {}
      locations: {}
      status: {}
      teachers: {}
    warn "couldn't read cached data"
  process.nextTick enrichData

Server

  app = express()
  app.use express.static "#{__dirname}/public"
  server = app.listen config.apiserver.port
  console.log "starting server on port: #{config.apiserver.port}"

REST server

  app.use (req, res, next) ->

no caching, if server through cdn

    res.header "Cache-Control", "public, max-age=0"

CORS

    res.header "Access-Control-Allow-Origin", "*"

no need to tell the world what server software we are running, - security best practise

    res.removeHeader "X-Powered-By"
    next()

  app.all "/stop-server", (req,res)->
    res.end "ok, exiting"
    setImmediate -> process.exit 0

  nextEvent = (agentOrLocation) ->
    events = data.next[agentOrLocation]
    return null if !events
    events.pop() while events.length && (events[events.length - 1] < getDateTime())
    events[events.length - 1]

  defRest = (name, member) ->
    app.all "/next/:id", (req, res) ->
      event = nextEvent req.params.id
      res.json(if event then {event: event} else {})
      res.end()

    app.all "/#{name}/:id", (req, res) ->
      res.json data[member][req.params.id]
      res.end()
    app.all "/#{member}", (req, res) ->
      res.json Object.keys data[member]
      res.end()

  endpoints =
    teacher: "teachers"
    activity: "activities"
    group: "groups"
    location: "locations"
    event: "events"
    agent: "agents"

  updateStatus = (cb) ->
    fs.stat config.apiserver.cachefile, (err, stat) ->
      status.organismTime = getDateTime()
      status.lastDataUpdate = stat?.mtime
      status.eventDetails =
          count: data.eventList.length
          pos: data.eventPos
          first: data.eventList[0]
          next: data.eventList[data.eventPos + 1]
          last: data.eventList[data.eventList.length - 1]
      status.connections = clientCount
      cb? status
  updateStatus()

  app.all "/status", (req, res) ->
    updateStatus (result) ->
      res.json result
      res.end()

  app.all "/now/:kind/:id", (req, res) ->
    res.json (data[req.params.kind + "Now"] || {})[req.params.id] || {}
    res.end()

  app.all "/current-state", (req, res) ->
    res.json data.agentNow
    res.end()

  app.all "/client-config", (req, res) ->
    try
      file = __dirname + "/client-config/default.json"
      file = __dirname + "/client-config/user.json" if fs.existsSync(__dirname + "/client-config/user.json")
      res.json JSON.parse(fs.readFileSync(file, "utf8"))
    catch e
      res.json {}
    res.end()

  app.all "/arrivals", (req, res) ->
    arrivals (result) ->
      res.json result
      res.end()

  defRest name, member for name, member of endpoints

When getting a request to /update, write it to data.json

For example upload with: curl -X POST -H "Content-Type: application/json" -d @datafile.json http://localhost:7890/update

  app.all "/update", (req, res) ->
    handleUCCData req.body, -> res.end()
  app.use "/uccorg-update", (req, res, next) ->
    result = ""
    req.on "data", (data) ->
      result += data
    req.on "end", ->
      console.log "getting #{result.length} bytes"
      handleUCCData (JSON.parse result), -> res.end()

Push server

Setup

  bayeux = new faye.NodeAdapter
    mount: '/faye'
    timeout: 45

  bayeux.on "subscribe", (clientId, channel) ->
    console.log channel, typeof channel

  clientCount = 0

  bayeux.on "handshake", (clientId, channel) ->
    ++clientCount

  bayeux.on "disconnect", (clientId, channel) ->
    --clientCount

  bayeux.attach server

Events and event emitter

  filterEvent = (event) ->
    currentEvent = data.events[data.agentNow[event.agents[0]]?.event] || {}
    if eventType(event) == "random"

only emit random events if they are happening during an activity where they can occur

      return if not (eventType(currentEvent) in event.during)

remember the previous event, to be able to restore it, - but only if it isn't a random event

      if eventType(currentEvent) != "random"
        data.beforeRandom[event.agents[0]] = currentEvent.id

    if eventType(event) == "random-end"
      if (data.agentNow[event.agents[0]]?.event) != event.ends
        data.beforeRandom[event.agents[0]] = undefined
        return
      prevEvent = data.events[data.beforeRandom[event.agents[0]] ]
      if prevEvent and !(prevEvent.description == "roaming" && prevEvent.autoLeave)
        event.description = prevEvent.description
        event.location = prevEvent.location
        event.clonedId = prevEvent.id
      else
        event.description = "roaming"
        event.autoLeave = leaveTime(event.id)

      data.beforeRandom[event.agents[0]] = undefined

    if event.description == "away" and data.agentNow[event.agents[0]].activity != "roaming"

TODO also go away if doing random stuff

      return
    return event

  autoLeave = []
  nextAutoLeave = "0"
  addAutoLeave = (event) ->
    autoLeave.push event
    nextAutoLeave = event.autoLeave if event.autoLeave < nextAutoLeave
  handleAutoLeave = (now) ->
    events = []
    if nextAutoLeave <= now
      list = autoLeave
      autoLeave = []
      nextAutoLeave = "9999"
      for event in list
        if event.autoLeave <= now
          for agent in event.agents
            if data.agentNow[agent].event == event.id
              id = event.autoLeave + '_' + uniqueHash(agent + event.id) + '_away'
              data.events[id] =
                id: id
                description: "away"
                time: event.autoLeave
                agents: [agent]
              events.push id
        else
          addAutoLeave event
    events

  leaveTime = (t) ->
    prng = prand(hash(t))
    (new Date(new Date(t.slice(0,19)+'Z') - (- (generalSettings.minRoam + (generalSettings.maxRoam - generalSettings.minRoam) * prng.next())|0) * 60 * 1000)).toISOString().slice(0,19)

  emitEvent = (event) ->
    event = filterEvent event
    return if not event
    addAutoLeave(event) if event.autoLeave
    data.events[event.id] = event if !data.events[event.id]
    console.log getDateTime(), event.id, event.description, event.location
    updateState event
    bayeux.getClient().publish "/events", event

  eventEmitter = ->
    now = getDateTime()
    events = handleAutoLeave now
    while data.eventPos < data.eventList.length and data.eventList[data.eventPos] <= now
      events.push data.eventList[data.eventPos]
      ++data.eventPos
    events.sort()
    for event in events
      emitEvent data.events[event]

  setInterval eventEmitter, 100

Train arrival data from rejseplanen

Get data

  arrivalCache = []
  getArrivals = (d, cb) ->

    url = "http://xmlopen.rejseplanen.dk/bin/rest.exe/arrivalBoard" +
      "?id=8600683&date=#{d.getUTCDate()}.#{d.getUTCMonth() + 1}.#{String(d.getUTCFullYear()).slice(2)}&time=#{d.getUTCHours()}:#{d.getUTCMinutes()}"
    (require "request") url, (err, _, data) ->
      return cb(err) if err
      arrivalCache = []
      !data.replace /<Arrival name="(.*?)"[^>]*?type="(.*?)"[^>]*?time="(.*?)" date="(.*?)" [^>]*? origin="(.*)">/g, (_, name, type, time, date, origin) ->
        arrivalCache.push
          name: name
          type: type
          date: "20#{date.slice(6,8)}-#{date.slice(3,5)}-#{date.slice(0,2)}T#{time}:00"
          origin: origin
      cb null, arrivalCache

  arrivals = (cb) ->
    now = getDateTime()
    getArrivals (new Date(now)), (err, result) ->
      if err
        cb []
      else
        cb result

Emit events

  lastArrivalEmit = undefined

  doArrival = (arrival) ->
    agentId = arrival.name + " " + arrival.origin
    agent = data.agents[agentId]
    if !agent
      agent = data.agents[agentId] =
        id: agentId
        kind: "transport"
        name: arrival.name
        origin: arrival.origin
    location = data.locations[arrival.type]
    if !location
      location = data.locations[arrival.type] =
        id: arrival.type
        kind: "transport"
    emitEvent
      id: getDateTime() + agentId + " arrive"
      time: getDateTime()
      description: "transport arrival"
      agents: [agent.id]
      location: arrival.type
    sleep 2 + Math.random() * 60, ->
      emitEvent
        id: getDateTime() + agentId + " leave"
        time: getDateTime()
        description: "transport leaving"
        agents: [agent.id]

  arrivalEmitter = ->
    now = getDateTime().slice(0,-6) + "00"

    doEmit = (arrs) ->
      if now == lastArrivalEmit
        return setTimeout arrivalEmitter, 30000
      lastArrivalEmit = now
      if !arrs.length
        return setTimeout arrivalEmitter, 60*60*1000
      for arrival in arrs
        if arrival.date == now
          doArrival arrival
      setTimeout arrivalEmitter, 30000

    if !arrivalCache.length || now >= arrivalCache[arrivalCache.length - 1].date
      arrivals doEmit
    else
      doEmit arrivalCache

  arrivalEmitter() if not config.test

update global state (agents/events)

  updateState = (event) ->
    return if not event
    for agent in event.agents
      prevLocation = (data.agentNow[agent] || {}).location
      data.locationNow[prevLocation].agents = data.locationNow[prevLocation].agents.filter ( (a) -> a != agent) if prevLocation
      location = event.location
      if location
        data.locationNow[location] = data.locationNow[location] || {}
        data.locationNow[location].agents = data.locationNow[location].agents || []
        data.locationNow[location].agents.push agent
        data.locationNow[location].event = event.id
      data.agentNow[agent] = {}
      data.agentNow[agent].location = location if location
      data.agentNow[agent].activity = event.description if event.description
      data.agentNow[agent].event = event.id

  calendarData enrichData #{{{2

Test

  if config.test
    testResult = ""
    testLog = (args...)->
      testResult += (JSON.stringify([args...]) + "\n").replace(/("id":"2015[^_]*)[^"]*/, '"id":"some-id')
    testDone = ->
      fs.writeFileSync config.test.outfile, testResult if config.test.outfile
      process.exit()



    testStart = config.test.startDate
    testEnd = config.test.endDate

Factor by which the time will run by during the test

    testSpeed = config.test.xTime

Rest test

    restTest = ->
      restTest = -> undefined
      console.log "restTest", getDateTime()

      url = "http://localhost:#{config.apiserver.port}/"
      restTestRequest = (id) -> (done) ->
        request url + id, (err, req, data) ->
          testLog id, JSON.parse data
          done()
      async.series [
        restTestRequest "now/location/Brikserum C.125"
        restTestRequest "now/group/49"
        restTestRequest "now/teacher/23"
        restTestRequest "now/location/C.284"
        restTestRequest "group/49"
        restTestRequest "teacher/23"
        restTestRequest "location/C.208"
        restTestRequest "activity/99009"
      ]
      undefined

Mock getDateTime,

Date corresponds to the test data set, and a clock that runs very fast

    testTime = + (new Date testStart)
    getDateTime = -> testTime
    setTimeout ( ->
      startTime = Date.now()
      getDateTime = -> (new Date(testTime + (Date.now() - startTime) * testSpeed)).toISOString() 
    ), 3000

run the test - current test client just emits "/events" back as "/test"

    bayeux.getClient().subscribe "/events", (message) ->
      testLog "event", message
    setInterval (->
      if config.test.restTestTime && getDateTime() >= config.test.restTestTime
        restTest()
      if getDateTime() >= testEnd
        testLog "testDone"
        testDone()
    ), 100000 / testSpeed

sendUpdate data, -> undefined


Autogenerated README.md, edit uccorg-backend.coffee to update repos