line |
stmt |
bran |
cond |
sub |
pod |
time |
code |
1
|
|
|
|
|
|
|
package Data::Chronicle; |
2
|
2
|
|
|
2
|
|
6
|
use strict; |
|
2
|
|
|
|
|
2
|
|
|
2
|
|
|
|
|
66
|
|
3
|
2
|
|
|
2
|
|
6
|
use warnings; |
|
2
|
|
|
|
|
2
|
|
|
2
|
|
|
|
|
93
|
|
4
|
|
|
|
|
|
|
|
5
|
|
|
|
|
|
|
=head1 NAME |
6
|
|
|
|
|
|
|
|
7
|
|
|
|
|
|
|
Data::Chronicle - Chronicle storage system |
8
|
|
|
|
|
|
|
|
9
|
|
|
|
|
|
|
=cut |
10
|
|
|
|
|
|
|
|
11
|
|
|
|
|
|
|
our $VERSION = '0.15'; |
12
|
|
|
|
|
|
|
|
13
|
|
|
|
|
|
|
=head1 DESCRIPTION |
14
|
|
|
|
|
|
|
|
15
|
|
|
|
|
|
|
This package contains two modules (Reader and Writer) which can be used to store and retrieve information |
16
|
|
|
|
|
|
|
on an efficient storage with below properties: |
17
|
|
|
|
|
|
|
|
18
|
|
|
|
|
|
|
=head2 Timeliness |
19
|
|
|
|
|
|
|
|
20
|
|
|
|
|
|
|
It is assumed that data to be stored are time-based meaning they change over time and the latest version is most important for us. |
21
|
|
|
|
|
|
|
|
22
|
|
|
|
|
|
|
=head2 Efficient |
23
|
|
|
|
|
|
|
|
24
|
|
|
|
|
|
|
The module uses Redis cache to provide efficient data storage and retrieval. |
25
|
|
|
|
|
|
|
|
26
|
|
|
|
|
|
|
=head2 Persistent |
27
|
|
|
|
|
|
|
|
28
|
|
|
|
|
|
|
In addition to caching every incoming data, it is also stored in PostgreSQL for future retrieval. |
29
|
|
|
|
|
|
|
|
30
|
|
|
|
|
|
|
=head2 Transparent |
31
|
|
|
|
|
|
|
|
32
|
|
|
|
|
|
|
This modules hides all the details about caching, database structure and ... from developer. He only needs to call a method |
33
|
|
|
|
|
|
|
to save data and another method to retrieve it. All the underlying complexities are handled by the module. |
34
|
|
|
|
|
|
|
|
35
|
|
|
|
|
|
|
Note that you will need to pass `cache_writer`, `cache_reader` and `db_handle` to the `Data::Chronicle::Reader/Writer` modules. These three arguments, provide access to your Redis and PostgreSQL which will be used by Chronicle modules. |
36
|
|
|
|
|
|
|
|
37
|
|
|
|
|
|
|
`cache_writer` and `cache_reader` should be to be able to get/set given data under given key (both of type string). `db_handle` should be capable to store and retrieve data with `category`,`name` in addition to the timestamp of data insertion. So it should be able to retrieve data for a specific timestamp, category and name. Category, name and data are all string. This can easily be achieved by defining a table in you database containing these columns: `timestamp, category, name, value`. |
38
|
|
|
|
|
|
|
|
39
|
|
|
|
|
|
|
=head1 METHODS |
40
|
|
|
|
|
|
|
|
41
|
|
|
|
|
|
|
There are four important methods this module provides: |
42
|
|
|
|
|
|
|
|
43
|
|
|
|
|
|
|
=head2 L<Data::Chronicle::Writer/set> |
44
|
|
|
|
|
|
|
|
45
|
|
|
|
|
|
|
Given a category, name and value stores the JSONified value in Redis and PostgreSQL database under "category::name" group and also stores current |
46
|
|
|
|
|
|
|
system time as the timestamp for the data (Which can be used for future retrieval if we want to get data as of a specific time). Note that the value |
47
|
|
|
|
|
|
|
MUST be either hash-ref or array-ref. |
48
|
|
|
|
|
|
|
|
49
|
|
|
|
|
|
|
$writer->set("category1", "name1", "value1"); |
50
|
|
|
|
|
|
|
$writer->set("category1", "name2", "value2", Date::Utility->new("2016-08-01 00:06:00")); |
51
|
|
|
|
|
|
|
|
52
|
|
|
|
|
|
|
=head2 L<Data::Chronicle::Reader/get> |
53
|
|
|
|
|
|
|
|
54
|
|
|
|
|
|
|
Given a category and name returns the latest version of the data according to current Redis cache |
55
|
|
|
|
|
|
|
|
56
|
|
|
|
|
|
|
my $value1 = $reader->get("category1, "name1"); #value1 |
57
|
|
|
|
|
|
|
|
58
|
|
|
|
|
|
|
=head2 L<Data::Chronicle::Reader/get_for> |
59
|
|
|
|
|
|
|
|
60
|
|
|
|
|
|
|
Given a category, name and timestamp returns version of data under "category::name" as of the given date (using a DB lookup). |
61
|
|
|
|
|
|
|
|
62
|
|
|
|
|
|
|
my $some_old_data = $reader->get_for("category1", "name2", Date::Utility->new("2016-08-01 00:06:00")); |
63
|
|
|
|
|
|
|
|
64
|
|
|
|
|
|
|
|
65
|
|
|
|
|
|
|
=head2 L<Data::Chronicle::Reader/get_for_period> |
66
|
|
|
|
|
|
|
|
67
|
|
|
|
|
|
|
Given a category, name, start_timestamp and end_timestamp returns an array-ref containing all data stored between given period for the given "category::name" (using a DB lookup). |
68
|
|
|
|
|
|
|
|
69
|
|
|
|
|
|
|
my $arrayref = $reader->get_for_period("category1", "name2", Date::Utility->new("2015-08-01 00:06:00"), Date::Utility->new("2015-08-01 00:06:00")); |
70
|
|
|
|
|
|
|
|
71
|
|
|
|
|
|
|
=head1 Examples |
72
|
|
|
|
|
|
|
|
73
|
|
|
|
|
|
|
my $d = get_some_log_data(); |
74
|
|
|
|
|
|
|
|
75
|
|
|
|
|
|
|
my $chronicle_w = Data::Chronicle::Writer->new( |
76
|
|
|
|
|
|
|
cache_writer => $writer, |
77
|
|
|
|
|
|
|
db_handle => $dbh); |
78
|
|
|
|
|
|
|
|
79
|
|
|
|
|
|
|
my $chronicle_r = Data::Chronicle::Reader->new( |
80
|
|
|
|
|
|
|
cache_reader => $reader, |
81
|
|
|
|
|
|
|
db_handle => $dbh); |
82
|
|
|
|
|
|
|
|
83
|
|
|
|
|
|
|
|
84
|
|
|
|
|
|
|
#store data into Chronicle - each time we call `set` it will also store |
85
|
|
|
|
|
|
|
#a copy of the data for historical data retrieval |
86
|
|
|
|
|
|
|
$chronicle_w->set("log_files", "syslog", $d); |
87
|
|
|
|
|
|
|
|
88
|
|
|
|
|
|
|
#retrieve latest data stored for syslog under log_files category |
89
|
|
|
|
|
|
|
my $dt = $chronicle_r->get("log_files", "syslog"); |
90
|
|
|
|
|
|
|
|
91
|
|
|
|
|
|
|
#find historical data for `syslog` at given point in time |
92
|
|
|
|
|
|
|
my $some_old_data = $chronicle_r->get_for("log_files", "syslog", $epoch1); |
93
|
|
|
|
|
|
|
|
94
|
|
|
|
|
|
|
=cut |
95
|
|
|
|
|
|
|
|
96
|
|
|
|
|
|
|
1; |