cankerwort[m]: no

hello.

tx JSON: termbin.com/ypkk

The CLSAG element is an array. It has only 1 index in this txn (e.g CLSAG[0])

When can there be multiple objects in the CLSAG array?

anyone?

luigi1111w, Thanks anyway for merging my CCS. This will help in a different dimension.

Is Monero's current LMDB configured to allow duplicate keys with different values?

Depends which table.

Where is the table-specific setting for that in the code?

or is it just enforced by flags in the  mdb_cursor_put calls?

I see this for example:

#define MDB_APPEND 0x20000

#define MDB_APPENDDUP 0x40000

why, for example, do we store the block info like

result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&zerokval, &val, MDB_APPENDDUP);

when the description for appendup says:

Store multiple data items in one call. Only for #MDB_DUPFIXED.

grep lmdb_db_open in src/blockchain_db/lmdb/db_lmdb.cpp

ah got you

so the flags are set there?

The ones with DUPSORT allow duplicate keys.

thanks

what about storing the block info as i mentioned above

why MDB_APPENDDUP? The description says we're storing multiple data items, but it seems that we're only storing one piece

Are you sure you're not thinking of MDB_MULTIPLE ?

ah my mistake

thanks

What's your guy's thoughts on users providing hashpower as form of payment for a product/service?

For example an online game that gives you in-game currency in the background in exchange for background mining

That is not an example of your question

the example has in game currency as the payment for the hashpower service

I guess if you partnered with a game creator, you could have something like that though

but few people would take hashpower as a payment

because essentially it's a gamble on future volatility

(of the price of whatever they're mining)

But I guess if they really wanted the ingame currency and there was no easy way to get hold of it they might overlook that

I would like this to happen.

If you want to do something about this, please feel free.

hmmm. looks like randomx disabled_flags() isn't checked for FLAG_LARGE_PAGES

is mdb_set_dupsort   for sorting keys only?

Two things of interest: primo and... not sure what it's called, tevador made a layer above randomx that a browser can call/use.

tevador's randomx server, yah

repo.getmonero.org/selene/primo for primo

It works for apache (web server) and firefox, but could work with anything where you want payment in return for a service really.

An initial attempt to stabilize the functional_tests_rpc's mining test:

mj-xmr: the problem is that it is difficult to test if this does indeed fix it

hyc do you know why for txs_prubable_hash we define the dupsort like:

mdb_set_dupsort(txn, m_txs_prunable_hash, compare_uint64);

because when we do a put, we're putting in a hash

mdb_cursor_put(m_cur_txs_prunable_hash, &val_tx_id, &val_prunable_hash, MDB_APPEND);

I didn't write that code, dunno

mooo did th pruning stuff

marcus40: a val_tx_id is a 64bit uint

not a hash

I see, the key should be uint64, the hash should be compare_hash32

could be a bug

moneromooo: ^^

fixing it would require full reload of the blockchain db

this bug means only 8 of 32byts of hash are being compared, so collisions could be possible

dunno the significance of that, dunno what txs_prunable_hash is for

I have a question regarding RPC SSL, does anyone know if is there a reason that the wallet does not use the system ca list?

as I recall, the point is not to make nodes identifiable

using a public CA would defeat that purpose

selsta: I have reproduced the problem by setting the timeout artificially to 1, and I got the same message.

mj-xmr: guess our best bet will be to merge it and wait if it shows up again

I also observed the CPU load, and indeed, the RX takes all the cores on initialization, just like xmrig miner. If this power is taken up by other processes, it's very natural for the timeout to be too small.

marcus40: #7374

yes

that's right yeah....i thought it should be hash32

I'm going to dump my current DB and see if the change makes any difference on reimport

do you mind if I PR it

I am trying to contribute to Monero more

wdy mean, it is a PR already

lol...

and it's not acceptable until we know the effect

impact

About to publish a new blog post about how to contribute, would love any feedback (specifically on the "filing issues" and "for developers" sections) before I push it: paste.debian.net/1185090

Don't want to cause more harm than good 🙂

PR already because you swooped in

lol

never mind :')

like I said, the PR isn't acceptable until we know what the impact will be

if there have been no collisions so far, then we're ok

alright

have to check with moneromooo what the significance of this really is

@hyc

when we do mdb_set_dupsort

that is for sorting keys only right?

set_compare is for keys

set_dupsort is for dup values

on your commit why have you put two entries?

There only needs to be one

both are needed

hmm.

I see, the DB is created with MDB_INTEGERKEY so set_compare is not needed

and the set_compare on txs_prunable is also not needed

why does that preclude it from needing a compare?

MDB_INTEGERKEY already means compare the key as an integer

just because it's the native format?

ok

got you

hyc if i'm going to store say a std::string consisting of the concatenation of three std::strings , do I need to convert to a C string before putting it in the db for the string compare to work properly?

sorry i meant if i have a string like that for BOTH the key and the value

you certainly cannot store std::string objects directly

yeah, but is c_str the way to go?

LMDB doesn't do string compares

it does set-of-bytes-of-a-given-length compares

BlockchainLMDB::compare_string

( i know thats what's going on under the hood)

but it's there, so would it make sense to just use that as the compare & dupsort?

also you said that the dupsort's are for values

but we're storing structures of block data here:

  MDB_val_set(val, bi);

  result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&zerokval, &val, MDB_APPENDDUP);

bi is a structure of block data

but we have   mdb_set_dupsort(txn, m_block_info, compare_uint64);

The data can be anything, but the key is uint64_t.

That table doesn't need to have duplicate keys I think.

So dupsort can be, er, whatever hte other one is.

compare IIRC

but hyc said that dupsort is for keys

sorry, for values

and it takes a compare function as an argument

Using the system CA list would mean any random CA on the system could sign a cert the wallet would silently accept (at first approximation, it might not do so if we use IPs directly).

If you want to check the cert, whitelisting it is much safer. Then you know it's the right one, not one some random CA told you is OK.

But vtnerd added the ability to use system CAs later on.

MDB_INTEGERKEY bypasses the compare funcs. I didn't know that :/

dupsort is for values but it implies compare IIRC. hyc ?

dupsort only sets the comparator for the values

doesn't touch anything for the keys

yeah so that's what I'm saying

block info has values that are blobs of mixed data

but the comparator is uint64_t

what gives

OK, so it should be setting compare, not dupsort. Do you want to PR that marcus40 ?

The first element of the block info structure is a uint64_t. The key happens to be empty.

Setting compare or just remocing the useless dupsort. I don't mind either.

ok, but just checking with you first

MDB_val_set(val_bi, height);

  result = mdb_cursor_get(m_cur_block_info, (MDB_val *)&zerokval, &val_bi, MDB_GET_BOTH);

  if (result == MDB_NOTFOUND)

  {

    throw0(BLOCK_DNE(std::string("Attempt to set cumulative difficulty from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- difficulty not in db").c_str()));

  }

  else if (result)

    throw0(DB_ERROR("Error attempting to set a cumulative difficulty"));

  mdb_block_info *result_bi = (mdb_block_info *)val_bi.mv_data;

  mdb_block_info bi;

  bi.bi_height = result_bi->bi_height;

  bi.bi_timestamp = result_bi->bi_timestamp;

  bi.bi_coins = result_bi->bi_coins;

  bi.bi_weight = result_bi->bi_weight;

  //bi.bi_diff_lo = diff; // TODO

  bi.bi_hash = result_bi->bi_hash;

  MDB_val_set(val, bi);

  result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&val_bi, &val, MDB_CURRENT);

here it's using the height as the key , and the entire info as the value

Don't spam here. Put this on a paste ste please.

sorry

paste.debian.net is nice.

ok i will in future

I guess here MDB_INTEGERKEY is accounting for the integer key

MDB_CURRENT means it'll add to what was found already.

is it not bad that we dont have a dupsort?

I assume the size of the value is ignored here.

This one's always append, it's not sorted.

Well, it could be I guess since it's sorted by height...

in other places, we only have result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&zerokval, &val, MDB_APPENDDUP);

no key this time?

A placeholder key looks like.

Well, real key, but empty.

So AFAICT it's not used in practice, the value is added at the end.

I'm getting a feeling I should let hyc address these things as he's the expert there

no it is... if there is no key, constant 'zerokey' concatenated with the 'data' becomes the key to access the data

however what i'm saying is some calls to put (m_cur_block_info

are using a placeholder key and some aren't

is that just because there are multiple types of data being stored in the same kvs?

In that table, no. Only the structure.

I think I understand what might be happening

But beings as we have no compator for the structure, that is prone to corruption right?

we need a dupsort (tx, ......, compare_struct)?

well compare_block_info

I don't fully understand why yet, but it appears that

(m_cur_block_info, (MDB_val *)&zerokval, &result, MDB_GET_BOTH);

where result is the height

allows you to pull a record created via:

result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&val_bi, &val, MDB_CURRENT);

where val_bi is a height and val is the block data

I guess it just means that (m_cur_block_info, (MDB_val *)&zerokval, &result, MDB_GET_BOTH);  is equivalent to looking up by height

@hyc

if compare() is being used for comparing key in application vs key in database

and dupsort is being used to compare values in application vs values in db,     *when dupsort is enabled*

what about when dupsort isn't enabled? where is our value comparator then

or am I missing the purpose of dupsort

everything else in there is correct

so whatever you're looking at, you've probably overlooked something

but what is used for value vs value comparison when we *dont* have dupsort enabled

please go read the LMDB docs

i did

it doesn't say

then use your brain

in a key value store, normally a key has only one value

since there is only one value, there is no need for a value comparator

but i thought the purpose of the comparator was to check the application value thats searched for, versus what comes out of the db

that's not what a key value store does

you give it a key to search for, it give you the corresponding value - whatever it is

there is no comparison check on values in a normal k/v

but LMDB supports multiple sorted values per key, in dupsort tables.

ONLY in dupsort tables does it do value comparisons, and need a value comparator. which is what the set_dupsort function provides.

this is why the get option is called GET_BOTH - that means check both the key and the value

if you don't use GET_BOTH, the value is not checked.

Right I see

thanks very much I get it now

I have one question then pertaining to monero

for the block_info table, we're storing a data structure. why is the comparator a uint64_t

the value comparator

shouldn't there be a custom comparator for the block info structure

mooo already answered you. the first field of the struct is a uint64

the rest of the fields are irrelevant for matching purposes

but that will laways be the same

the first field is the height of the block

(which is also the key)

there is only one block_info per block height so what's the problem?

there is only one block for a given height

what about alt blocks that have the same height

they aren't in this table

doesn't it store those in there too? orphans

if there is only one block per height it shouldn't be dupsort in the first place then right?

Am I ok to remove the dupsort( ... ..... ., uint64_t compare)

wrong and don't touch

so no duplicate values... why dupsort?

Read the DB schema description more carefully

everything is already explained

Please try and be a bit less harsh, you're sounding like fireice sometimes :P

They're all duplicates. That table has a single key (that empty one I mentioned above).

this is all in the source already, these are stupid questions.

this is all *documented* already

It takes a while to grok everything.

marcus40: read all of this comment github.com/monero-project/monero/bl…blockchain_db/lmdb/db_lmdb.cpp#L189

And the empty key trick is not something you expect.

that's why it is explicitly documented there

and it's a cardinal sin to suggest changes to code that you haven't read thoroughly

What can I say. I'm a sinner...

well i suggested one earlier and it turned out to be right

Do you mean it was just luck ? :)

lol

you quickly lapped that up hyc eh

haha

but fair enough.... new to this part of the code

I feel like the comments are often poor

Not very clear/ thorough

but thanks and see you

running an import now with the dupsort patch to see how different the DBs are

It should not matter, this call was pointless IIRC ?

That was for a non DUPSORT table, right ?

txs_prunable_hash is a dupsort table tho

is there only 1 hash per ID?

I guess that would be the case

ok now I see

there's no bug

Oh bloody hell with that Copyright.

#7334

the only thing I'd point out is that the txs_prunable_hash table is created as dupsort but doesn't actually have any duplicate values

so no bug, just a little misleading

Ah. I don't actually rememver how it works now...

it's a plain jane key/value table. key is txid, value is tx hash

OK, then dropping the flag seems like a good idea, yes.

it could have been setup as dupfixed to save some space

Can't recall.

but then it would have to do the key prefix trick etc. which it doesn't do

at this point we can't really change it. LMDB would complain on dbi_open that the flags are mismatched

(when opening an existing DB)

Oh I see.

vtnerd: around ?

yes

I'm replacing vector with span as you suggested, and I'm getting this:

The ctor I'm aiming for (const T*, size_t) can't be used due to safe_conversion being false.

But the struct is nothing fancy, just a struct { unsigned char[32]; }

Do you see why offhand ?

Line 4800 is now: return bulletproof_PROVE(epee::span<rct::key>(&sv, 1), rct::keyV(1, gamma));

(the second one will be a span too after this is sorted)

it looks like const-correctness for at least some of the errors. `epee::span<const rct::key>(&sv, 1)` -> mismatched types `rct::key [N]` and `const rct::key*`"

well it wouldn't use that overload, but thats the first set of errors

Oh right. That's one thing that confused me earlier, vector doesn't work like this. Thanks.

oh I think the other errors are saying it cannot call the other overload

Indeed, it was just that. I should have worked it out -_- Thanks.

FIY, you want to know why you are getting those daily messages? Cults NEED to control information. That's why people like lh1008 get savagly attacked (also called disconnection) for even talking to me. Break the information embargo, break the cult. Who would sign up to Scientology if they knew from the get-go that is is about a sci-fi alien overlord? Who would sign up to MAGA if they knew what a loser

Trump is?

You don't sign up to MAGA to do a failed Viking LARP and some time off in club fed. You sign up because you feel like shit and they promise to give you self-respect. It is a total lie of course, you are still a joke even when cosplaying a Viking. Just like Monero's promise of privacy is a lie.

OMFG, fuck this waste of time for just two extra fucking cycles -_-

Fucking spans. They can go in later.

it wasn't an easy transition? and its probably way more than two cycles given the allocation

Sure, but it's one alloc for the whole of PROVE or VERIFY. De minimis.

And no, not easy. span is missing a ctor from const U* AFAICT, but adding it did not help either.

I mean, it might be easy when you're used to C++ery, but it's wasting my time.

And I'll be back in an hour when the tests have run with the remaining changes.

how to debug this monero-wallet-rpc error? Error calling gettransactions daemon RPC: r 1, status <error>

why does it need another constructor? none of those types have bases for conversion

It has a ctor with U * const src_ptr, but none with const U *src_ptr

So if you have a const object to start with, &object will be const, and so can't bind AFAIK.

Anyway, I reset and redid the other changes. If you want to switch to span later, you're welcome (also in bulletproofs, which has vectors too).

But it's just taking me way too much time for not much gain.