Commit Graph

174 Commits

Author SHA1 Message Date
Chris Eager
13053da97f Add Twilio Verify experiment to AccountController 2021-04-26 14:31:52 -05:00
Chris Eager
5974328d9c Ensure accounts are deleted after batch migration; store migration failures for later processing 2021-04-21 11:13:07 -05:00
Chris Eager
a472774734 Add threadpoool to increase Accounts → Dynamo migration throughput 2021-04-21 11:13:07 -05:00
Chris Eager
59bbd0c43c Add Accounts DynamoDB
* Add additional test cases to AccountsTest
* Migrate AccountsManagerTest to JUnit 5
* Add AccountsDynamoDbConfiguration
* Add Account.dynamoDbMigrationversion
* Add DynamicAccountsDynamoDbMigrationConfiguration
* Add AccountsDynamoDb to AccountsManager
* Add AccountsDynamoDbMigrator
2021-04-16 14:24:24 -05:00
Jon Chambers
2cb788ceb7 Delete secure backups when deleting accounts. 2021-04-01 11:30:48 -04:00
Jon Chambers
ab4e94edab Revert "Revert "Delete data in the storage service when deleting accounts.""
This reverts commit 91fc0fd623.
2021-03-30 17:17:32 -04:00
Chris Eager
ad6c271f9d Add dynamic configuration for signup captcha by country code 2021-03-30 10:22:02 -05:00
Chris Eager
83aa59f4dd Align messages_cluster client name 2021-03-29 14:51:26 -05:00
Jon Chambers
7e29ed1cc7 Block attempts to set wallet addresses from unsupported countries. 2021-03-26 12:21:11 -04:00
Jon Chambers
a7bad20eae Revert "Add an admin command for printing a subset of account data."
This reverts commit a288b9df8e.
2021-03-22 10:32:38 -04:00
Jon Chambers
089b6b1644 Retry attempts to get messages after a delay; close connections after a finite number of retries. 2021-03-22 10:32:25 -04:00
Jon Chambers
a288b9df8e Add an admin command for printing a subset of account data. 2021-03-18 12:43:12 -04:00
Chris Eager
5dc8086968 Migrate Twilio numbers from static to dynamic configuration 2021-03-16 12:53:51 -05:00
Jon Chambers
fde1b49729 Introduce a set of gauges for various network statistics as reported by nstat. 2021-03-09 11:45:53 -05:00
brock-signal
1faedd3870 Return Retry-After time to clients when they are rate limited (#421)
* Return Retry-After time to clients when they are rate limited

* Update based on feedback

- New exception type that is mapped differently
- Always report time until allowed on rate limits
- Consume and transform into a differnt exception if we think it will be
  allowed later
2021-03-05 10:23:03 -07:00
Jon Chambers
eb6fe11da1 Add tools to decline messages from senders meeting specific conditions 2021-02-24 18:09:26 -05:00
Jon Chambers
06ca5f14fc Record the age of accounts that send unsealed-sender messages. 2021-02-23 12:16:45 -05:00
Moxie Marlinspike
71f0aab2c6 Actually start the currency managed lifecycle 2021-02-17 14:28:08 -08:00
Moxie Marlinspike
2dbab70c8c Create utility endpoint for currency conversion 2021-02-16 11:57:34 -08:00
Jon Chambers
5f49772ca6 Control enforcement of unsealed sender rate limits via dynamic configuration. 2021-02-12 15:43:05 -05:00
Jon Chambers
ff448950ed Collapse the feature flag system into the dynamic config system. 2021-02-12 15:23:34 -05:00
Jon Chambers
ad32555cc9 Drop the old push scheduler Redis singleton. 2021-02-11 14:01:11 -05:00
Ehren Kret
be8a1acca9 Remove message database from the codebase (#395)
* Remove message database from the codebase

* Remove unused ExperimentEnrollmentManager in test

* Be more stylish
2021-02-11 10:50:03 -06:00
Jon Chambers
71510a8199 Decommission the old directory cache. 2021-02-10 11:02:11 -05:00
Jon Chambers
2f105ed0a4 Add support for remote client deprecation 2021-02-09 12:24:44 -05:00
Jon Chambers
6c0de89de8 Remove scary message-throttling feature flags. 2021-02-08 11:55:01 -05:00
Jon Chambers
04728ea4bc Drop the old Postgres-based pre-key store. 2021-02-08 11:40:43 -05:00
Jon Chambers
3298db8683 Split traffic between the Postgres and Dynamo pre-key stores. 2021-02-04 15:18:53 -05:00
Jon Chambers
d4d9403829 Add a Dynamo-backed key store. 2021-02-04 15:18:53 -05:00
Jon Chambers
b413f665d8 Make DynamicConfigurationManager not-really-Managed. 2021-02-04 11:33:33 -05:00
Moxie Marlinspike
5e1a572bd8 Track impact of unsealed sender rate limits (#374) 2021-02-03 08:58:47 -08:00
Ehren Kret
0dcb4b645c Build Dynamo DB backed Message Store (#358)
* Work in progress...

* Finish first pass draft of MessagesDynamoDb

* Use begins_with everywhere for destination device id

* Remove now unused methods

* First basic test built

* Add another test case

* Remove comment

* Verify more of the message contents

* Ensure all methods are tested

* Integrate MessagesDynamoDb into the MessagesManager

This change plugs the MessagesDynamoDb class into the live serving
flow in MessagesManager.

Tests are not yet as comprehensive for this big a change as they
should be, but they now compile and pass so checkpointing here with a
commit.

* Put DynamoDB before RDBS when deleting specific messages

* Extract method

* Make aws sdk version into a property

* Rename clientBuilder

* Discard messages with no GUID

* Unify batching logic into one function

* Comment on the source of the value in this constant

* Inline method

* Variable name swizzle

* Add timers to all public methods

* Add missing return statements

* Reject messages that are too large with response code 413

* Add configuration to control dynamo DB timeouts

* Set server timestamp from the ReceiptSender

* Change to shorter key names to optimize IOPS

* Fix tests broken by changing column names

* Fix broken copyright template output

* Remove copyright template error text

* Add experiments to control use of dynamo and rds in message storage

* Specify instance profile credentials for the dynamic configuration manager

* Use property for aws sdk version

* Switch dynamo to instance profile credentials

* Add metrics to the batch write loop

* Use placeholders in logging
2021-02-03 10:03:19 -06:00
Moxie Marlinspike
92f6a79e1f Add a dynamic configuration manager 2021-02-01 11:01:58 -05:00
Jon Chambers
46c800b8b7 Smoosh request logging tasks together rather than having one task for each direction. 2021-01-25 14:58:15 -05:00
Jon Chambers
f10be893ce Drop the old feature flag controller. 2021-01-25 14:55:57 -05:00
Jon Chambers
c606c1664f Add admin tasks for listing, setting, and deleting feature flags. 2021-01-25 14:55:57 -05:00
Jon Chambers
943a5d1036 Shard push scheduling cache 2021-01-19 15:50:12 -05:00
Jon Chambers
0a23ce870a Allow message persisters to be disabled by a feature flag. 2021-01-17 11:13:12 -05:00
Jon Chambers
c355ef8d53 Reduce the message cache thread pool size. 2021-01-16 11:15:25 -05:00
Jon Chambers
59a0fd0799 Embiggen message cluster thread pool. 2021-01-16 02:57:04 -05:00
Jon Chambers
00b5cfcf17 Allow the client presence manager to use an entirely separate cluster. 2021-01-16 02:57:04 -05:00
Jon Chambers
9e342f253d Use the same client for inserts and reads in the message cache cluster. 2021-01-16 01:50:40 -05:00
Jon Chambers
20c48b6bb2 Expand message-related thread pools to 1 thread per shard. 2021-01-16 01:50:40 -05:00
Jon Chambers
4f9e7bb572 Separate Lettuce thread pools. 2021-01-16 01:18:05 -05:00
Jon Chambers
a57ce1dd17 Add machinery to allow a percentage of message sends to succeed. 2021-01-15 17:05:16 -05:00
Jon Chambers
93ae4d1ee6 Move the client presence manager to its own breaker. 2021-01-15 13:51:39 -05:00
Jon Chambers
9c53d818f4 Use separate clusters for message cache read/write operations. 2021-01-15 13:51:39 -05:00
Jon Chambers
67ed035b36 Retry serializable key transactions. 2021-01-13 17:38:29 -05:00
Jon Chambers
2e01da5ec1 Add a task to enable/disable accelerated crawling. 2021-01-11 19:29:18 -05:00
Ehren Kret
86ccaa52a5 Allow configuration of multiple directory account crawler listeners (#325)
* Allow configuration of multiple directory account crawler listeners

Only one should update the local redis directory. This one is marked
with replicationPrimary true. The others in the list only serve to
issue replication requests over to CDS replication load balancers.

* Update one more metric name
2021-01-10 17:11:02 -06:00