内置时间变量
1.总述#
需求背景#
用户希望在写代码时，对时间的格式要求千变万化,已有的Linkis自定义变量目前还不足以支撑这些需求。另外,已有的时间运算-1有些表示减一个月,有些则是减一天,用户很容易混淆
目标#
- 其他日期内置变量都是相对run_date计算出来
- 支持Pattern格式时间且用户可以自行随意指定
- 支持±y/±M/±d/±H等等
Pattern格式对照表:
| Letter | Date or Time Component | Presentation | Examples |
|---|---|---|---|
| G | Era designator | Text | AD |
| y | Year | Year | 1996; 96 |
| Y | Week year | Year | 2009; 09 |
| M | Month in year | Month | July; Jul; 07 |
| w | Week in year | Number | 27 |
| W | Week in month | Number | 2 |
| D | Day in year | Number | 189 |
| d | Day in month | Number | 10 |
| F | Day of week in month | Number | 2 |
| E | Day name in week | Text | Tuesday; Tue |
| e | Day number of week (1 = Monday, …, 7 = Sunday) | Number | 1 |
| a | Am/pm marker | Text | PM |
| H | Hour in day (0-23) | Number | 0 |
| k | Hour in day (1-24) | Number | 24 |
| K | Hour in am/pm (0-11) | Number | 0 |
| h | Hour in am/pm (1-12) | Number | 12 |
| m | Minute in hour | Number | 30 |
| s | Second in minute | Number | 55 |
| S | Millisecond | Number | 978 |
| z | Time zone | General time zone | Pacific Standard Time; PST; GMT-08:00 |
| Z | Time zone | RFC 822 time zone | -0800 |
| X | Time zone | ISO 8601 time zone | -08; -0800; -08:00 |
2. 总体设计#
总体设计和技术架构参照Linkis自定义变量
3. 功能介绍#
- Linkis支持的变量类型分为自定义变量(不做赘述)和系统内置变量，自定变量日期支持+-。
- 其中+-是先对linkis内置参数run_date进行运算,然后将%之前的pattern字段进行替换,非Pattern字符不支持运算替换。
3.1 内置变量举例#
可以根据自己的喜好/业务实际情况定义需要动态渲染的参数
| variable | result |
|---|---|
| &{yyyy-01-01} | 2021-01-01 |
| &{yyyy-01-01%-2y} | 2019-01-01 |
| &{yyyy-MM-01%-2M} | 2021-02-01 |
| &{yyyy-MM-dd%-2d} | 2021-03-31 |
| &{yyyy MM ----- HH%-1H} | 2021 04 ----- 14 |
| &{yyyyMMdd%-1d} | 20210401 |
| &{yyyyMM01%-1M} | 20210301 |
| &{HH%-1H} | 14 |
3.2 自定义变量使用实例#
- 例子1: sql
SELECT * FROM hive.tmp.fund_nav_historiesWHERE dt <= DATE_FORMAT(DATE_ADD('day', -1, DATE(Date_parse('&{yyyyMMdd%-1d}', '%Y%m%d'))), '%Y%m%d')渲染后
SELECT * FROM hive.tmp.fund_nav_historiesWHERE dt <= DATE_FORMAT(DATE_ADD('day', -1, DATE(Date_parse('20220705', '%Y%m%d'))), '%Y%m%d')- 例子2: shell
aws s3 ls s3://***/ads/tmp/dws_member_active_detail_d_20210601_20211231/pt=&{yyyyMMdd%-1d}/渲染后
aws s3 ls s3://***/ads/tmp/dws_member_active_detail_d_20210601_20211231/pt=20220705/- 例子3: datax json
{ "job": { "setting": { "speed": { "channel": 1 } }, "content": [ { "reader": { "name": "s3reader", "parameter": { "bucket": "**************", "path": [ "ads/tmp/ccass_tm_announcements/&{yyyyMMdd%-1d}/" ], "stored": "parquet", "compression": "NONE", "column": [ { "index": 0, "type": "int" }, { "index": 1, "type": "string", "constant": "&{yyyyMMdd%-1d}" } ] } }, "writer": { "name": "streamwriter", "parameter": { "print": true } } } ] }}渲染后
{ "job": { "setting": { "speed": { "channel": 1 } }, "content": [ { "reader": { "name": "s3reader", "parameter": { "bucket": "**************", "path": [ "ads/tmp/ccass_tm_announcements/20220705/" ], "stored": "parquet", "compression": "NONE", "column": [ { "index": 0, "type": "int" }, { "index": 1, "type": "string", "constant": "20220705" } ] } }, "writer": { "name": "streamwriter", "parameter": { "print": true } } } ] }}- 例子4:python
print(&{yyyyMMdd%-1d})渲染后
20220705